AFIT/GLM/LAL/97J-2 


TRAINING  MANAGEMENT  INFORMATION  SYSTEM 
OF  THE  DEFENSE  INSTITUTE  OF  SECURTTYASSISTANCE 
MANAGEMENT:  USER  SATISFACTION 
AS  A  MEASURE  OF  ITS  EFFECTIVENESS 

THESIS 

Paulo  R.de  Oliveira  Ruy,  Major,  BAF 

AnT/GLM/LAL/97J-2 


Approved  for  public  release;  distribution  unlimited 

19970630  078 


DTIC  QUAIiTTY  mSPECTED  L 


The  views  expressed  in  this  thesis  are  those  of  the  author  and  do  not  reflect  the  official 
policy  or  position  of  the  Department  of  Defense  or  the  U.S.  Government  or  the  Brazilian 
Government. 


AnT/GLM/LAL/97J-2 


TRAINING  MANAGEMENT  INFORMATION  SYSTEM  OF  THE  DEFENSE 
INSTITUTE  OF  SECURITY  ASSISTANCE  MANAGEMENT:  USER  SATISFACTION 
AS  A  MEASURE  OF  ITS  EFFECTIVENESS 


THESIS 


Presented  to  the  Faculty  of  the  School  of  Logistics 
and  Acquisition  Management  of  the  Air  Force  Institute  of  Technology 

Air  University 

Air  Education  and  Training  Command 
In  Partial  Fulfillment  of  the  Requirements  for  the 
Degree  of  Master  of  Science  in  Logistics  Management 


Paulo  R.  de  Oliveira  Ruy,  B.S. 
Major,  BAF 

June  1997 


Approved  for  public  release;  distribution  unlimited 


Acknowledgments 


I  would  like  to  recognize  my  appreciation  to  my  faculty  advisors,  Dr.  Craig  M. 
Brandt  and  Major  William  L.  Scott,  for  their  guidance  and  suggestions  dispensed  during 
this  thesis  effort.  Their  insight  and  experience  were  decisive  to  the  success  of  this 
research  project.  Also,  I  would  like  to  express  my  gratitude  to  my  sponsor,  Charles  E. 
Collins,  Associate  Professor  at  the  Defense  Institute  of  Security  Assistance  Management 
(DISAM),  for  his  assistance  in  explaining  details  of  the  Training  Management  System 
(TMS)  as  well  as  in  sending  the  survey  by  e-mail  to  the  Security  Agency  Organizations 
(SAO)  around  the  world.  In  addition,  I  really  appreciated  the  people  of  each  S  AO  for 
their  consideration  in  taking  their  precious  time  to  answer  the  survey. 

I  can  not  miss  this  opportunity  to  demonstrate  my  sincere  gratitude  to  Aimette 
Rowell  Robb,  Chief  of  International  Military  Student  Officer,  for  all  assistance  and 
advise  to  solve  the  most  common  issues  inherent  to  any  international  student  in  a  foreign 
country. 

Finally,  and  most  importantly,  I  wish  to  recognize  the  patience,  devotion  and 
understanding  demonstrated  by  my  wife  Vera,  my  son  Daniel,  and  my  daughter  Raquel 
along  this  almost  past  twenty-nine  months  dedicated  to  the  graduate  studies.  Certainly, 
without  their  support  and  encouragement  this  endeavor  would  have  been  much  harder. 


Paulo  R.  O.  Ruy 


Table  of  Contents 


Page 

Acknowledgments . ii 

List  of  Figures . vi 

List  of  Tables . vii 

Abstract . viii 

1.  Introduction . 1 

Background . 1 

Problem  Statement . 2 

Research  Questions . 3 

Methodology . 4 

Scope . 4 

Summary . 4 

n.  Literature  Review . 6 

Introduction . 6 

Model  for  System  Effectiveness . 6 

System  Quality . 7 

Information  Quality . 10 

Information  Use . 1 1 

User  Satisfaction . 12 

End-User  Computing  Satisfaction . 13 

User  Information  Satisfaction . .13 

VaUdation . 15 

Construct  VaUdity . 15 

EDP  Staff  and  Services . 15 

Information  Product . 16 

Knowledge  and  Involvement . 16 

Convergent  Validity . 17 

Individual  Impact . 17 

Organizational  Impact . 18 

The  Training  Management  System  (TMS) . 19 


iii 


in.  Methodology 


Page 

,...25 


Introduction . 25 

Research  Design . 26 

The  Sample  Population . 27 

The  Data  Collection  Procedure . 28 

The  Survey  Instrument . 28 

Content . 28 

Rating  Scale . 28 

Reliability . 29 

Methods  of  Analysis . 32 

IV.  Results  and  Analysis . 33 


Introduction . 33 

Demographic  Data . 33 

Years  of  Use . 33 

Civilian  or  Military . 34 

Version  3.1  or  4.0 . 34 

Score  Results . 34 

T-test  for  Total  Satisfaction  and  Each  Factor . 37 

Regression  Analysis . 39 

V.  Conclusions  and  Recomendations . 42 

Conclusions . 42 

Question  1 . 42 

Question  2 . 43 

Question  3 . 43 

Question  4 . 44 

Question  5 . 44 

Recommendations  for  TMS  Staff. . 45 

Recommendations  for  Future  Research . 46 

Summary . 46 


IV 


Page 


Appendix  A:  Survey . 48 

Appendix  B:  Raw  Data . 51 

Appendix  C:  Correlation  Analysis . 55 

Appendix  D:  Correlation  Analysis . 72 

Appendix  E:  Correlation  Analysis . 78 

Appendix  F:  t-test . 79 

Appendix  G:  Regression  Analysis . 83 

Bibliography . 86 

Vita . 89 


V 


List  of  Figures 


Figure  Page 

1.  Information  System  Success  Model . 7 

2.  S.A.  Training  Conununity  Use  of  the  SAN . 21 

3.  IMS  Software  in  the  SAN . 22 

4.  Linear  Regression  Plot . 41 


VI 


List  of  Tables 


Table  Page 

1.  Reliability  Scores . 30 

2.  Means,  Standard  Deviations,  Intercorrelations,  and  Coefficient  Alpha 

Reliability  Estimates  for  the  Study’s  Variables . 31 

3.  Years  of  Using  the  System . 34 

4.  Average  Satisfaction  Scores  for  Standard  Questionnaire . 36 

5.  Average  Satisfaction  Scores  for  Added  Questions . 37 

6.  P- values  for  paired  t-test . 38 

7.  T-test  for  Two-Sample  Assuming  Unequal  Variances . 39 

8.  Regression  Analysis  year  x  average  satisfaction . 40 


vii 


AnT/GLM/LAL/97J-2 


Abstract 


The  purpose  of  this  study  was  to  evaluate  the  effectiveness  of  the  Training 
Management  System  (TMS)  installed  in  the  Security  Assistance  Organizations  around  the 
world.  User  satisfaction  was  measured  as  an  indicator  of  the  system’s  effectiveness. 

In  order  to  provide  an  objective  measurement  of  the  system  effectiveness,  the 
following  research  questions  were  addressed:  (1)  What  is  the  system  effectiveness 
regarding  the  level  of  product  quahty  provided  by  TMS  ?  (2)  What  is  the  level  of 
involvement  and  knowledge  of  TMS  user  related  to  the  information  services  function? 

(3)  What  is  the  level  of  user  perceived  satisfaction  with  the  staff  and  services  provided 
by  support  people  of  TMS  ?  (4)  What  is  the  perceived  difference  in  levels  of  satisfaction 
between  mihtary  and  civilian  for  each  of  the  questions  1, 2  and  3  above?  (5)  What  is  the 
impact  of  experience  with  the  system  on  questions  1  to  3  above? 

User  satisfaction  was  determined  to  be  the  best  possible  measure  of  system 
effectiveness  and  it  was  measured  by  administering  a  user  satisfaction  survey.  The  data 
gathered  from  this  survey  was  analyzed  and  that  analysis  provided  the  basis  for 
concluding  that  TMS  was  meeting  the  users’  needs,  but  that  the  system  effectiveness 
could  be  improved  by  providing  training.  Recommendations  were  offered  to  the  TMS 
staff  support  and  suggestions  for  further  research  were  also  given. 
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MANAGEMENT  INFORMATION  SYSTEM  OF  THE  DEFENSE  INSTITUTE  OF 


SECURITY  ASSISTANCE  MANAGEMENT;  USER  SATISFACTION  AS  A 
MEASURE  OF  ITS  EFFECTIVENESS 

I.  Introduction 

Background 

The  overseas  Security  Assistance  Organization  (SAO)  is  the  U.  S.  Government 
office,  normally  located  in  the  American  Embassy,  that  is  responsible  for  the  overseas 
management  of  the  U.  S.  security  assistance  program.  Security  assistance  is  a  group  of 
programs  authorized  by  the  Foreign  Assistance  Act  of  1961  and  the  Arms  Export  Control 
Act  (as  amended)  by  which  the  United  States  provides  defense  articles  or  services 
(including  military  training)  to  other  countries.  In  the  case  of  the  international  military 
training  program,  the  SAO  is  totally  responsible  for  the  identification  of  all  host-country 
training  requirements,  the  requesting  of  that  training  from  U.  S.  military  services,  and  the 
execution  and  implementation  of  the  training. 

The  Security  Assistance  Network  (SAN)  is  a  particular  type  of  Information 
System  that  contains  among  various  items  a  database  used  in  the  management  of  the 
international  military  training  program.  This  database  is  called  Training  Management 
System  (TMS)  which  intended  use  is  to  provide  the  training  program  manager  in  the 
overseas  Security  Assistance  Office  (SAO)  information  related  to  the  training  program. 

In  this  researeh  TMS  will  be  evaluated  regarding  its  effectiveness  as  measured  under  the 
view  of  user  satisfaction. 
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The  development  of  such  an  information  system  involves  the  spending  of 
economical  and  human  resources  which  are  becoming  more  and  more  scarce.  Therefore 
it  is  important  to  know  if  this  investment  is  providing  the  expected  return  to  the 
Department  of  Defense  (DoD).  One  indirect  way  to  know  if  this  return  is  adequate  is  by 
measuring  its  degree  of  effectiveness. 

Problem  Statement 

The  training  management  system  (TMS)  provides  a  modem  database  management 
system  that  allows  the  S  AO  to  effectively  manage  its  training  program  which  includes 
generating  training  requests,  tracking,  programmed  and  scheduled  training,  recording  all 
required  informational  data  on  the  student  trainees,  producing  required  Defense 
Department  documentation  on  the  students,  and  entering  the  students  into  training  in  the 
U.  S.  Prior  to  the  advent  of  the  TMS  system,  this  was  accomplished  via  a  computer  print 
out  (three  different  versions  from  the  three  different  military  services)  mailed  worldwide 
once  a  month  with  delivery  times  of  up  to  a  month. 

However,  the  problem  that  arises  is  related  to  the  fact  that  regardless  of  how  well 
the  system  is  operating  technically  or  improved  its  features,  its  level  of  effectiveness  may 
or  may  not  be  favorably  when  viewed  by  the  user. 

Therefore,  the  problem  is  to  measure  the  system  effectiveness  as  a  dependent 
variable  of  user  satisfaction,  information  quality,  and  system  quality.  Also  it  is  necessary 
to  compare  different  versions  of  the  system,  and  provide  data  to  decision  makers  of  the 
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TMS  staff.  This  will  enable  them  to  better  understand  the  view  of  the  user,  and  as  a 
result,  help  them  to  improve  the  product  and  quality  of  system  support  in  specific  areas. 

In  other  words,  the  purpose  of  this  work  is  to  measure  the  degree  of  TMS  success 
in  order  to  provide  data  to  the  system’s  manager  and  make  him  able  to  improve  the 
quahty  of  services  and  products  offered  by  TMS.  This  practice  will  make  the  manager 
able  to  implement  changes  based  on  the  perceptions  of  the  user  by  hstening  to  his/her 
voice  and  make  the  necessary  decisions. 

Research  Questions 

In  order  to  provide  an  objective  measurement  of  system  effectiveness  for  the 
Training  Management  System,  the  following  research  questions  must  be  answered. 

1 .  What  is  the  system  effectiveness  regarding  to  the  level  of  product  quality  provided  by 
TMS? 

2.  What  is  the  level  of  involvement  and  knowledge  of  TMS’  user  related  to  the 
information  services  function? 

3.  What  is  the  level  of  user  perceived  satisfaction  with  the  staff  and  services  provided  by 
support  people  of  TMS  ? 

4.  What  is  the  perceived  difference  in  levels  of  satisfaction  between  military  and  civilian 
for  each  of  the  questions  1,2  and  3  above? 

5.  What  is  the  impact  of  experience  with  the  system  on  questions  1  to  3  above? 


3 


Methodology 

The  methodology  adopted  in  this  research  is  a  survey  of  user  information 
satisfaction  based  in  a  validated  self-administered  questionnaire.  This  questionnaire  was 
sent  by  e-mail  to  many  S  AOs  around  the  world  in  order  to  capture  users  opinion  about 
TMS.  The  respondents  sent  back  the  answers  by  e-mail  or  by  fax.  After  this,  the  raw 
data  were  processed  in  order  to  draw  conclusions  about  TMS’  effectiveness. 

Scope 

The  results  of  this  research  are  not  generalizable  to  other  types  of  information 
systems  beyond  the  TMS.  Even  though  the  questionnaire  is  validated,  and  its  reliability 
was  replicated  and  evaluated  by  using  the  survey  data,  the  appUcability  and  the 
recommendations  draw  from  the  analysis  concerning  the  system  effectiveness  are  specific 
to  TMS. 

Also,  the  findings  of  this  research  will  be  analyzed  focusing  on  detecting  areas  of 
the  system  that  requires  improvement  in  order  to  increase  system  effectiveness.  It  is  out 
the  scope  of  this  research  to  indicate  specific  ways  of  how  to  achieve  the  goals  of  the 
recommended  management  actions. 

Summary 

The  remaining  chapters  of  this  research  are  divided  as  follow.  Chapter  11  has  the 
objective  to  review  the  literature  related  to  the  topic  of  this  research,  beginning  with  a 
broad  model  of  system  effectiveness  and  narrowing  it  to  the  specific  areas  of  research. 
Chapter  III  describes  in  depth  the  methodology  adopted  in  this  research,  presenting  the 
design  of  the  sample  frame,  the  method  of  data  collection,  and  the  evaluation  of  the 
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survey  instrument  respecting  to  its  validity.  Chapter  IV  analyses  the  results  of  this 
research  presenting  the  processed  raw  data  in  form  of  tables.  This  chapter  also,  describes 
the  statistical  tests  utilized  and  presents  results  of  regression  analysis.  Finally,  Chapter  V 
addresses  the  conclusions  based  on  the  findings  previously  analyzed  in  Chapter  IV  and 
provides  recommendations  for  management  action  as  well  as  for  future  research. 


5 


II.  Literature  Review 


Introduction 

This  chapter  describes  the  concept  of  information  system  effectiveness  from  a  perspective 
of  total  quality  management  and  the  importance  of  listening  to  the  voice  of  the  customer 
as  a  measure  of  system  effectiveness.  Also,  this  chapter  provides  a  review  of  the 
validated  survey  instruments  used  to  measure  user  satisfaction  as  an  indicator  of  system 
effectiveness.  Finally,  a  description  of  the  system  under  investigation  is  provided 
concerning  its  functions,  products  and  users. 

Model  for  System  Effectiveness 

Much  research  has  been  done  in  the  field  of  information  system  effectiveness. 
DeLone  and  McLean  (1992)  has  organized  this  research  into  a  comprehensive  model  of 
information  system  success.  The  model  proposed  presents  information  system 
effectiveness  or  success  composed  of  six  interrelated  dimensions:  system  quality, 
information  quality,  use,  user  satisfaction,  individual  impact,  and  organizational  impact. 
The  idea  of  interdependency  among  the  six  dimensions  is  showed  in  Figure  1. 

The  model  of  information  system  success  is  viewed  as  a  process  construct.  In  this 
process  one  should  consider  the  idea  of  time  as  weU  as  the  causal  relationship  among  the 
six  categories.  For  example  use  and  user  satisfaction  are  both  effected  by  system  quality 
and  information  quality  when  considered  singularly  and  jointly.  Also,  use  and  user 
satisfaction  interact  between  themselves  and  both  have  influence  over  individual  impact. 
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Finally,  this  latter  dimension  has  some  effect  over  organizational  impact.  Each  of  these 


dimensions  are  discussed  below. 


Figure  1.  Information  System  Success  Model  (DeLone  and  McLean,  1992). 

System  Quality  .  Total  quality  management  (TQM)  is  a  concept  whose  focus  is 
on  the  customer.  Regardless  of  whether  the  customer  is  internal  or  external  to  the 
company,  his  demands  and  satisfaction  must  be  met.  Therefore,  the  first  element  of 
organizational  culture  affected  by  TQM,  is  focus.  There  is  no  question  that  a  firm 
customer  focus  brings  competitive  advantage.  In  addition,  TQM  takes  the  view  that 
profits  follow  quahty,  not  the  other  way  around.  Poor  quality  can  be  seen  and  will  be 
rejected.  Fail  to  satisfy  a  customer’s  quality  demands  even  once,  and  the  company  might 
lose  that  customer  possibly  forever.  As  a  result,  profit  cannot,  in  reality,  be  “maximized” 
if  customers  choose  a  competitor’s  product.  If  enough  customers  refuse  to  buy  the 
company’s  offerings,  then  profit  will  be  nonexistent. 

Now,  the  issue  is  how  can  we  bring  the  concept  of  customer  satisfaction  to  the 
field  of  Information  Systems  (IS).  In  other  words,  what  does  TQM  mean  to  IS 
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considering  the  aspect  of  customer  satisfaction?  Under  TQM,  it  will  not  be  enough  to 
deliver  defect-free  systems  according  to  user  requirements.  The  IS  professional  has  to 
look  beyond  the  user  to  the  user’s  customer  to  be  sure  the  system  is  actually  “doing  the 
right  thing.”  In  other  words,  the  true  customer  for  the  system  has  to  be  identified  and  his 
requirements  considered  (Ward,  1994).  In  doing  that,  a  quality  system  is  able  to  deliver 
defect-free  output  and  to  meet  the  expectations  of  all  potential  customers.  The  literature 
has  presented  many  examples  of  signals  emitted  by  the  customers  trying  to  communicate 
their  complaints,  in  product  output  or  services  of  IS  (Palmer,  1991).  Therefore,  IS 
professionals  need  to  consider  the  user  involvement  beginning  with  the  early 
requirements  phase,  through  design,  implementation,  and  maintenance  (Seim,  1980).  The 
latter  is  as  important  as  the  early  phase  because  the  environment  is  dynamic  and  it  is 
necessary  to  continually  monitor  system  quality  through  customer  satisfaction. 

Wang  and  Kon  (1997)  developed  a  framework  for  Total  Data  Quality 
Management  in  which  the  concept  of  continuous  measurement,  analysis,  and 
improvement  is  introduced.  The  first  concept  relates  to  the  importance  of  a  clear 
understanding  of  the  multiple  dimensions  encompassing  data  quality,  because  the 
production  of  quality  data  can  be  limited  without  this  understanding.  Second,  it  is 
necessary  to  know  how  analysis  of  data  quality  impacts  business,  for  example,  by  relating 
data  quality  to  sales,  customer  satisfaction,  and  profitability.  Finally,  the  concept  of 
continuous  improvement  encompasses  three  dimensions:  business  redesign,  in  order  to 
minimize  the  occurrence  of  data  errors;  motivation,  which  focuses  on  how  rewards  can 
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improve  data  quality;  new  technologies,  which  emphasizes  aspects  of  improvement  in 
techniques  of  data  entry. 

Considering  measures  of  system  quality  there  is  one  early  work  developed  by 
Swanson  (1974)  who  used  several  system  quality  items  to  measure  MIS  appreciation 
among  user  managers.  His  items  included  the  reliability  of  computer  system,  on-line 
response  time,  the  ease  of  terminal  use,  and  so  forth.  Hamilton  and  Chervany  (1981)  also 
proposed  data  currency,  response  time,  turnaround  time,  data  accuracy,  reliability, 
completeness,  system  flexibility,  and  ease  of  use  as  part  of  a  framework  to  measure 
system  quality. 

In  a  more  recent  work,  Pitt  et  al  (1995)  developed  an  instrument  to  measure 
service  quality.  The  questionnaire  consists  of  22  scales  each  one  ranging  from  1  through 
7,  which  presented  in  a  factor  analysis  five  components  for  the  construct  service  quality: 
tangibles,  which  assesses  physical  facilities,  equipment,  and  appearance  of  personnel; 
reliabihty,  which  is  related  to  the  ability  to  perform  the  promised  service  dependably  and 
accurately;  responsiveness,  which  means  the  wilhngness  to  help  customers  and  provide 
prompt  service;  assurance,  related  to  the  knowledge  and  courtesy  of  employees  and  their 
ability  to  inspire  trust  and  confidence;  and  empathy,  which  relates  to  caring, 
individualized  attention  the  service  provider  gives  its  customers. 

Among  the  several  practical  applications  using  empirical  measures  of  system 
quality,  there  is  one  field  study  performed  by  Srinivasan  (1985)  in  which  he  evaluated  the 
computer-based  modeling  systems  of  29  firms  accessing  their  system  quality  by  using  a 
measure  which  included  response  time,  system  rehability,  and  system  accessibility. 
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Information  Quality.  This  dimension  focuses  on  the  quality  of  the  information 


system  output.  In  other  words,  it  is  related  to  the  quality  of  the  information  that  the 
system  produces,  primarily  in  the  form  of  reports.  For  example,  DeLone  and  McLean 
(1992)  have  emphasized  the  role  of  information  quality  and  user  satisfaction  as  two  major 
dimensions  for  evaluating  the  success  of  information  systems.  These  two  dimensions 
generally  include  some  data  quality  attributes,  such  as  information  accuracy,  output 
timeliness,  precision,  reUability,  currency,  completeness,  and  relevancy  (Ives,  B.  et  al, 
1983  and  Bailey  and  Pearson,  1983).  In  this  same  line  of  research,  Olson  and  Lucas 
(1982)  proposed  report  appearance  and  accuracy  as  measures  of  information  quality  in 
office  automation  information  systems.  Also,  under  the  broad  area  of  “User  Information 
Satisfaction”  livari  and  Koskela  (1987)  developed  a  satisfaction  measure  including  three 
information  quality  constructs:  “informativeness”  which  consists  of  relevance, 
comprehensiveness,  recentness,  accuracy,  and  credibility;  “accessibility”  which  consists 
of  convenience,  timeliness,  and  interpretability;  and  “adaptability”. 

In  a  more  recent  work,  Wang  and  Strong  (1996)  developed  a  framework  based  on 
data  collected  from  data  consumers,  considering  the  aspects  of  data  quality  that  are  more 
important  to  them.  Their  findings  showed  four  data  quaUty  categories  each  one 
consisting  of  specific  attributes.  The  first  category  consists  of  accuracy,  objectivity, 
believability,  and  reputation  and  denotes  that  data  have  quahty  in  their  own  right.  The 
second  consists  of  value-added,  relevancy,  timeliness,  completeness,  and  appropriate 
amount  of  data  and  highhghts  the  aspect  that  data  quality  must  be  considered  within  the 
context  of  the  task  at  hand.  The  third  one  consists  of  interpretabiUty,  ease  of 
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understanding,  representational  consistency,  and  concise  representation.  Finally,  the 
fourth  includes  accessibihty  and  access  security  with  these  latter  two  categories 
emphasizing  the  importance  of  the  role  of  systems  play  in  data  quality  issues. 

Some  researchers  applied  information  quality  measures  in  their  field  studies.  For 
example.  Miller  and  Doyle  (1987)  performed  a  field  study  in  an  overall  I/S  including  21 
financial  firms.  They  surveyed  276  user  managers  by  using  a  measure  that  covered 
attributes  of  completeness  of  information,  accuracy  of  information,  relevancy  of  reports 
and  timeliness  of  report.  They  found  that  the  overall  IS  success  is  correlated  with  the 
relationship  between  perceived  performance  and  importance.  Another  finding  was  that 
there  is  a  mismatch  between  perceived  importance  and  performance  in  developing  more 
of  the  different  basic  systems  types. 

Information  Use.  System  usage,  the  utilization  of  information  technology  (IT)  by 
individuals,  groups  or  organizations,  is  a  core  variable  in  IS  research.  This  dimension 
focuses  on  the  use  of  information  system  reports  which  is  one  frequently  reported 
measures  of  the  success  of  an  information  system.  Under  the  conceptual  view  Ein-Dor 
and  Segev  (1978)  claimed  that  different  measures  of  computer  success  are  mutually 
interdependent  and  so  they  chose  system  use  as  the  primary  criterion  variable  for  their  l/S 
research  framework. 

An  objective  approach  to  measure  usage  is  computer-recorded  data.  It  can  be 
measured  for  example  as  the  number  of  computer  inquiries,  percentage  of  customer 
records  updated  annually,  connect  time  and  frequency  of  computer  sections,  number  of 
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information  reports  generated  by  the  system  and  the  frequency  of  user’s  access  and 
connect  time  along  with  the  number  of  light,  average,  and  heavy  users  (Srinivasan,  1985). 

In  addition  to  conceptual  studies,  this  measure  has  been  used  in  field  studies.  For 
example,  Adams  et.  al.  (1992)  used  this  measure  in  their  studies  of  e-mail  and  voice-mail 
considering  the  number  of  messages  as  the  system  usage  measure.  Also,  in  their 
evaluation  of  WordPerfect  and  Harvard  Graphics  they  used  extent  of  use  as  their  measure 
of  system  usage.  The  other  dimension  to  be  reviewed  is  the  User  Satisfaction,  as  popular 
as  the  Use  dimension. 

User  Satisfaction.  Ives  et  al  (1983)  define  User  Information  Satisfaction  as  the 
extent  to  which  users  believe  the  information  system  available  to  them  meets  their 
information  requirements.  There  has  been  much  attention  paid  to  the  concept  of  user 
satisfaction  as  a  measure  of  effectiveness  of  information  systems  and  the  productivity  of 
the  system  users. 

Determining  the  effectiveness  of  an  organization’s  information  services  function 
is  a  crucial  management  concern.  This  determination  may  lead  to  the  decision  to 
restructure,  or  even  out-source  the  information  services  function  (ISF).  Developing 
measures  of  effectiveness  has  been  a  focus  of  MIS  research  and  the  most  commonly  used 
measures  of  effectiveness  within  the  MIS  field  are  user’s  perceptions  of  satisfaction 
(Gatian,  1984).  There  has  been  significant  research  into  determining  user  satisfaction.  In 
general,  two  types  of  user  satisfaction  instruments  have  been  developed:  the  End-User 
Computing  Satisfaction  (EUCS)  and  User  Information  Satisfaction  (UIS). 
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End-User  Computing  Satisfaction.  Instruments  to  measure  End-User 


Computing  Satisfaction  were  developed  as  a  result  of  the  increase  in  personal  computing 
in  the  1980s  and  1990s.  Doll  and  Torkzadeh  (1988)  to  investigated  the  measurement  of 
the  user  satisfaction  construct  in  relation  to  the  growing  field  of  end-user  computing 
(EUC).  They  created  a  12-item  instrument  based  on  five  components  of  user  satisfaction: 
content,  accuracy,  format,  ease  of  use,  and  timeliness.  Etezadi-Amoli  and  Farhoomand 
(1996)  have  developed  a  measure  of  end  user  computing  satisfaction  (EUCS).  The 
instrument  was  developed  around  six  elements  of  EUCS  including  documentation,  ease 
of  use,  functionality  of  system,  quality  of  output,  support,  and  security.  The  research 
supported  the  idea  that  user  satisfaction  was  useful  in  determining  the  user  performance 
(Etezadi-Amoli  and  Farhoomand,  1996).  However,  while  their  measure  attempts  to 
capture  the  satisfaction  of  end-users  with  respect  to  a  specific  information  systems 
application,  it  does  not  include  a  direct  assessment  of  end-user  support  services  or,  more 
generally,  of  the  ISF.  In  order  to  gauge  the  products  and  services  of  a  firm’s  information 
services  function,  a  second  user  satisfaction  instrument.  User  Information  Satisfaction 
(UIS),  has  been  frequently  employed. 

User  Information  Satisfaction.  Early  instruments  measuring  User 
Information  primarily  focused  on  the  quality  of  information  product  attributes  such  as 
relevance,  timeliness,  accuracy,  and  format.  Bailey  and  Pearson  (1983)  created  one  of  the 
earhest  tools  for  measuring  computer  user  satisfaction,  a  39  item  user  satisfaction  survey. 
Ives  et  al  (1983)  and  Baroudi  and  Orlikowski  (1986)  continued  their  work  and  in  a 
replication  study  sought  to  shorten  the  instrument  to  include  22  and  13  items. 
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respectively,  which  they  asserted  still  effectively  measured  the  user  acceptance  construct 
and  reinforced  its  validity. 

The  instrument  developed  by  Bailey  and  Pearson  (1983)  broadened  the  user 
satisfaction  measurement  to  focus  on  additional  dimensions  of  ISF  activities  including 
training,  documentation,  and  staff  communication.  This  instrument  has  been  the  focus  of 
a  series  of  studies  which  identified  its  major  dimensions  as:  (1)  Quality  of  Information 
Products,  which  focuses  on  the  product  or  technical  quality  of  information  systems 
delivered  by  the  ISF;  (2)  Level  of  User’s  Knowledge  and  Involvement,  which  connotes 
the  proactive  posture  of  the  users  to  participate  with  the  information  services  function  or 
in  systems  development;  and,  (3)  Attitude  towards  Electronic  Data  Processing  Staff  and 
Services,  which  focuses  on  an  assessment  of  perceived  satisfaction  with  the  staff  and 
services  of  the  ISF.  Thus  the  evolved  and  empirically  derived  instrument  attempts  to 
evaluate  both  product  and  service  dimensions  of  IS  customer  satisfaction.  That  is,  it 
intends  to  assess  not  only  what  the  ISF  delivers,  but  also  how  and  under  what  conditions 
this  is  done.  Among  the  three  dimensions  of  UIS,  the  Quality  of  Information  Products 
dimension  includes  technical  assessment  of  the  content  and  presentation  of  product.  The 
other  two  dimensions,  especially  the  attitude  towards  EDP  Staff  and  Services  dimension, 
clearly  capture  the  functional  attributes  about  services  and  user  relationships  with  ISF 
staff.  The  next  sub-section  will  describe  in  more  detail  the  survey  instrument  of  Baroudi 
and  Orlikowski  (1986)  in  terms  of  its  validation. 

Validation.  In  this  section  the  construct  validity  and  convergent  validity  of  the 
short  -form  Baroudi  and  Orlikowski’s  measure  instrument  are  described. 
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Construct  Validity.  It  is  established  by  showing  that  “the  measure  is  an 
appropriate  operational  definition  of  the  construct  it  purports  to  be  measuring”  (Stone, 
1978).  Baroudi  and  Orlikowski  (1988)  replicated  two  methods  used  before  by  other 
researchers.  The  first,  weaker  method  examines  the  correlation  between  each  scale  and 
the  total  UIS  score.  The  results  for  this  method  presented  the  13  correlations  ranging 
from  .35  to  .69  with  1 1  correlating  at  levels  greater  than  .50. 

The  second  method  uses  factor  analysis  which  allows  an  examination  of  the 
underlying  structure  of  the  measure.  The  results  showed  that  the  13  questions  were 
decomposed  into  three  groups: 

EDP  Staff  and  Services.  This  factor  is  related  to  respondents’ 
self-reported  assessment  of  the  attitude  and  responsiveness  of  the  EDP  staff  as  well  as  the 
quality  of  their  relationship  with  the  EDP  staff.  It  includes  the  following  scales  as 
showed  in  the  questionnaire  of  Appendix  A: 

Ql-  Relationship  with  the  DISAM  TMS  support  staff; 

Q2-  Processing  of  requests  for  changes  in  TMS; 

Q6-  Attitude  of  the  TMS  support  staff; 

Ql  1 -Communication  with  the  TMS  support  staff; 

Q  12-Time  required  for  TMS  4.0  development. 
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Information  Product.  This  factor  means  the  respondent  judgment 


respect  to  the  quality  of  output  delivered  by  the  information  system.  It  includes  the 
following  scales: 

Q7-  Reliability  of  output  information  provided  by  TMS; 

Q8-  Relevancy  of  output  information  provided  by  TMS; 

Q9-  Accuracy  of  output  information  provided  by  TMS; 

QIO-  Precision  of  output  information  provided  by  TMS; 

Q13-  Completeness  of  output  information  provided  by  TMS. 

Knowledge  and  Involvement.  This  factor  is  directly  related  to  the 
quality  of  training  provided  to  the  respondents,  their  understanding  of  the  system,  and 
their  participation  in  its  development.  It  includes  the  scales: 

Q2-  Processing  of  requests  for  changes  in  TMS; 

Q3-  Degree  of  TMS  training  provided  to  users; 

Q4-  User’s  understanding  of  the  TMS; 

Q5-  User’s  feelings  of  participation. 

This  replication  performed  by  Baroudi  and  Orlikowski  shows  that  all  but  one  of 
the  13  scales  loads  as  expected.  The  only  exception  is  question  2  that  loads  on  two 
factors:  EDP  staff  and  services,  and  knowledge  and  involvement.  This  is  aceptable  and 
lead  the  conclusion  that  “the  factor  structure  of  the  questionnaire  is  stable  and  provides 
strong  evidence  for  the  construct  validity  of  the  measure.”  (Baroudi  and  Orlikowoski, 
1988). 
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Convergent  Validity.  According  to  Stone,  “  the  extent  to  which  a  measure 
is  correlated  or  “agrees”  with  other  measures  of  the  same  construct  provides  evidence  for 
convergent  validity”  (Stone,  1978).  In  order  to  replicate  this  validity,  Baroudi  and 
Orlikowski  performed  interviews  of  user  satisfaction  within  five  organizations  and 
formed  two  groups  :  satisfied  and  dissatisfied.  Then  they  used  a  t-test  to  compare  the 
mean  score  for  each  group  and  found  that  there  was  a  significant  difference  between  the 
two  groups  a  p<  .001.  They  concluded  that  these  results  “provide  some  evidence  for  the 
measure’s  convergent  validity”  (Baroudi  and  Orlikowski,1988). 

Considering  the  multidimensional  characteristic  of  the  MIS  success  construct,  the 
research  performed  by  DeLone  and  McLean  (1992),  among  other  findings,  found  that  the 
user  satisfaction  measure  developed  by  Bailey  and  Pearson  (1983),  covers  three 
dimensions  of  the  six  proposed  in  the  model.  In  particular  the  Bailey  and  Pearson  (1983) 
measure  covers  the  dimensions  of  information  quality,  system  quality,  and  user 
satisfaction.  The  instrument  to  be  used  in  this  research  is  the  UIS  considering  its 
multidimensional  characteristics  of  including  service  quality  dimensions  and  relationships 
with  ISF  staff  in  addition  to  information  quality. 

Individual  Impact.  This  dimension  is  about  the  effect  of  information  on  the 
behavior  of  the  recipient.  The  impact  or  influence  of  information  can  be  viewed  for 
example  in  the  decision  context  if  the  user  has  a  better  understanding  of  the  decision,  or  if 
he/she  has  improved  the  productivity  of  the  decision-making  process,  or  has  changed  the 
decision  maker’s  perception  of  the  importance  or  usefulness  of  the  information  system. 
Also,  decision  effectiveness  can  be  defined  as  the  dependent  success  variable  including 
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dimensions  of  the  average  time  to  make  a  decision,  the  confidence  in  the  decision  made, 
and  the  number  of  reports  requested.  Therefore  one  method  of  measuring  FS  impact  is  to 
determine  whether  the  output  of  the  system  causes  the  decision  maker  to  change  his  or 
her  behavior. 

Organizational  Impact.  This  dimension  relates  to  the  effect  of  information  on 
organizational  performance.  This  idea  can  be  explained  considering  that  the  success  of 
the  MIS  department  is  reflected  in  the  extent  to  which  the  computer  is  applied  to  critical 
or  major  problem  areas  of  the  firm.  For  example  one  company  can  be  ranked  on  the  basis 
of  the  range  and  scope  of  its  computer  applications,  or  the  firm’s  abiUty  to  computerize 
high  complexity  applications. 

Considering  the  economics  view  a  firm  can  be  assessed  by  the  cost  reductions  and 
company  profits  realized  from  specific  user-developed  application  programs.  Following 
this  idea,  the  company  revenues  can  also  be  improved  by  computer-based  information 
systems.  In  this  case  it  should  be  measured  the  relationship  of  corporate  outcomes  such 
as  total  sales  and  return  on  investment  to  FS  inputs. 

With  the  corporate  “bottom-line”  in  mind,  it  should  be  considered  that  MIS 
effectiveness  be  determined  by  its  contribution  to  company  profits,  for  example  asking 
users  of  a  particular  I.S.  what  savings  were  realized  from  the  use  of  that  I.S.  and  what 
costs  were  incurred  by  using  it.  Therefore,  one  measure  of  organizational  performance 
which  might  be  appropriate  in  this  context  is  return  on  investment.  And  in  a  broader 
sense  one  could  measure  the  extent  to  which  an  office  information  system  contributed  to 
meeting  organizational  goals. 
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Given  the  significance  of  the  information  quality  and  the  importance  of  the  end- 
user  role  in  determining  that  quality,  this  thesis  will  research  system’s  effectiveness  and 
its  impact  on  user  information  satisfaction  with  the  specific  information  system,  the 
Training  Management  System.  Therefore  it  is  important  to  determine  the  degree  to  which 
the  measurement  of  User  Information  Satisfaction  (UIS)  provides  the  TMS  users  a 
product  that  fulfills  their  expectation  and  requirements.  This  measurement  will  be  useful 
in  detecting  areas  in  which  that  system  needs  improvement. 

The  Training  Management  System  (TMS) 

The  security  assistance  network  (SAN)  is  used  by  all  levels  of  Security  Assistance 
program  management  to  communicate  in  the  worldwide  Security  Assistance 
environment.  Thus,  the  SAN  is  used  by  the  overseas  Security  Assistance  Organization 
(SAO)  training  offices,  the  Unified  Command  security  assistance  training  offices,  the 
Defense  Security  Assistance  Agency  (DSAA)  training  managers,  the  Military  Department 
training  managers,  and  the  various  military  school  international  military  student  offices. 
Figure  1  provides  a  graphic  portrayal  of  this  network.  Data  for  the  various  programs, 
including  the  international  military  training  program,  are  provided  via  the  SAN  (SAN  I, 
1997). 

The  training  program  data  consists  of  the  Military  Articles  and  Services  List 
(MASL),  which  is  a  master  price  list  of  all  U.  S.  training  available  to  the  foreign 
customer.  It  includes  information  such  as  the  course  title,  duration,  price,  location, 
security  classification,  etc.  The  Standardized  Training  List  (STL)  is  the  actual  training 
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that  has  been  requested  and  approved  for  the  country  and  includes  both  financial  (cost 
information)  and  dates  of  the  training.  A  third  database  is  the  Location  Table  data  which 
simply  provides  information  on  the  training  location. 

Training  program  data  is  uploaded  by  the  military  training  management 
organizations  (Army,  Navy,  and  Air  Force)  to  the  SAN  and  where  it  is  consohdated  and 
made  available  via  controlled  access  to  the  overseas  training  manager.  This  is  portrayed 
in  Figure  2. 
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Figure  3.  TMS  Software  in  the  SAN  (DISAM,  1996). 


The  training  management  system  (TMS)  is  a  database  management  system  that 
assists  the  primary  user,  the  overseas  Security  Assistance  training  management  office,  in 
the  management  and  execution  of  the  intematioanal  military  training  program.  Program 
data  is  downloaded  from  the  security  assistance  network,  imported  to  the  TMS  system, 
and  used  to  generate  various  management  reports  that  facilitate  program  management. 
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TMS  provides  both  detail  and  summary  reports  on  the  country’s  training  program. 
The  summary  reports  allow  the  S  AO  training  manager  to  manage  the  country  training 
program  and  keep  it  within  the  authorized  State  Department  allocation  level.  The  detail 
reports  allow  the  S  AO  to  keep  track  of  all  the  training  that  has  been  requested  and  must 
be  entered  into  the  training  program  data  by  the  military  department  training  agencies  in 
the  United  States.  TMS  generates  reports  that  tell  the  SAO  training  manager  when  the 
next  student  is  to  be  given  an  English  Language  test  or  when  the  student  must  depart  for 
scheduled  training  in  the  U.  S.  TMS  generates  the  DD  Form  2285,  the  official 
Invitational  Travel  Order  which  is  a  legal  document  that  is  required  for  the  student  to 
enter  training. 

The  system  provides  a  means  to  generate  training  request  messages  to  the  military 
departments  that  result  in  required  changes  to  the  training  program.  TMS  extracts  data 
from  the  various  miUtary  department  provided  databases  such  as  the  MASL,  STL,  and 
Location  Table  databases,  combines  this  with  the  SAO  generated  student  information 
database,  and  then  produces  the  official  invitational  travel  order  (ITO)  required  by 
regulation  for  all  international  mihtary  students  to  enter  US  training(DISAM,  1996). 

Fielding  of  the  TMS  software  was  begun  in  1991-92  with  the  first  versions,  TMS 
1.0,  TMS  2.0,  etc.  The  first  fully  operational  version  of  the  software  was  fielded  as  TMS 
3.1  in  early  1994.  It  was  developed  by  a  civilian  contractor  using  FoxPro  and  could  only 
be  changed  by  having  the  contractor  accomplish  program  changes.  That  version  has 
proven  to  be  quite  reliable  and  continues  to  be  used  by  many  SAOs(SAN  11,1996).  TMS 
Version  4.0,  programmed  using  MS  Access  by  US  government  personnel,  was  first 
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fielded  worldwide  in  the  spring  of  1996.  Fielding  of  TMS  4.1  was  being  accomplished 
as  work  on  this  analysis  progressed.  A  final  edition  of  TMS  4. 1  will  be  fielded  in  about 
June/July  1997  after  all  problems  identified  with  TMS  4.0  have  been  solved.  For  these 
reasons  the  version  to  be  surveyed  is  the  4.0  and  not  4.1  (SAN  11, 1996). 

The  greatest  differences  between  the  various  versions  of  the  TMS  software  are 
found  between  the  TMS  3.1  and  4.X  versions.  TMS  3.1  was  “hard  coded”  using  Fox  Pro 
and  was  a  simple,  straight  forward  DOS  based  system.  TMS  4.0,  a  MS  Access 
application,  was  much  more  sophisticated  and  provide  for  greater  capabihty  for  the  user. 
For  example  one  capability  of  TMS  3.1  that  was  never  duplicated  in  TMS  4.X  was  a 
“custom  report”  function  that  allowed  the  user  to  formulate  his  own  unique  reports  on  the 
various  databases.  TMS  4.X  attempted  to  provide  a  very  wide  range  of  canned  reports 
that  would  meet  all  user  needs.  However,  as  it  was  the  first  version  of  the  4.X  series,  it 
definitely  had  more  “bugs”  in  it.  TMS  4.0  was  not  released  as  a  test  or  “beta”  version, 
but  rather  as  a  final  release.  Complete,  final  testing  had  not  effectively  been 
accomplished.  TMS  4.1  corrected  the  various  problems  and  added  features  that  had  been 
requested  by  users. 

The  Defense  Institute  of  Security  Assistance  Management  (DISAM)  is  responsible 
for  all  aspects  of  system  development,  fielding,  user  training,  and  system  support 
(Collins,  1997). 
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in.  Methodology 


Introduction 

The  most  common  way  pointed  by  the  literature  to  monitor  the  voice  of  the  customer  is 
through  surveys  of  user  information  satisfaction  (UIS)  (Scott,  1988).  Using  these  surveys 
as  a  gauge,  it  will  be  possible  to  measure  the  effectiveness  of  a  system  and  consequently 
its  “successes”  (Newcomer,  1991).  Besides  measuring  effectiveness  it  is  necessary  to 
know  if  we  are  addressing  the  issues  most  important  to  the  customer  to  provide  a 
complete  picture  of  user  satisfaction  (Richardson,  1994).  Much  research  has  been  done  in 
this  field  to  incorporate  attributes  of  quality  into  products  of  IS.  These  attributes, 
accuracy,  consistency,  conformity,  precision,  timeliness  and  usability,  were  measured 
using  validated  and  reliable  tools  (Mallach,  1988).  As  a  result,  many  problems  related  to 
usefulness  and  usableness  of  MIS  reports  can  be  minimized  (Rand,  1984).  In  order  to 
obtain  the  data  necessary  for  analysis,  this  research  will  rely  upon  a  survey  of  user 
satisfaction  considering  the  reasons  mentioned  above.  It  will  make  use  of  an  available 
and  already  validated  self-administered  questionnaire  (Baroudi  and  Orlikowski,  1988) 
that  will  be  sent  to  a  sample  of  users  in  many  S  AOs  around  the  world  in  order  to  capture 
their  opinions. 

The  researchers  validated  the  questionnaire  using  a  sample  of  358  employees 
representing  a  wide  variety  of  industries  including  banking,  insurance,  retailing,  and 
manufacturing,  with  an  average  of  12  employees  from  each  company. 

In  order  to  evaluate  the  construct  validity  the  researchers  used  factor  analysis 
method  which  showed  that  the  13  questions  were  decomposed  into  three  groups: 
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electronic  data  processing  (EDP)  staff  and  services,  information  product,  knowledge  and 
involvement.  The  researchers  evaluated  the  convergent  validity  by  performing  interviews 
of  user  satisfaction  within  five  organizations.  Two  groups  were  formed:  satisfied  and 
dissatisfied,  and  the  results  for  a  paired  t-test  for  the  mean  of  each  group  showed 
significant  difference  between  them.  Finally  the  rehability  was  determined  by  calculating 
Crombach’s  alpha  for  the  two  items  which  comprise  each  of  the  13  factors,  for  the  overall 
satisfaction,  and  for  each  of  the  three  factors.  All  the  reliabilities  were  above  .80,  which 
provided  evidence  that  the  questionnaire  is  internally  consistent  and  thus  reasonably  free 
of  measurement  error. 

The  current  research  will  extend  their  results  in  the  sense  that  the  organization 
utilized  is  a  not  for  profit  organization.  In  addition  the  system  under  study  is 
characterized  as  a  global  network,  which  is  evidenced  by  the  questionnaire  being 
responded  by  43  persons  of  different  countries  both  military  and  civiUan.  Also,  this 
research  will  evaluate  the  sensitivity  of  user  information  satisfaction  to  the  time  of  using 
the  information  product. 

The  elements  of  the  System  Effectiveness  model  to  be  covered  in  this  research 
are  information  quaUty,  system  quality,  and  user  satisfaction,  which  are  characteristics  of 
the  system  that  will  be  addressed  in  order  to  answer  the  research  questions. 

Research  Design 

The  purpose  of  the  survey  is  to  describe  situations  faced  by  the  user  of  TMS  in 
terms  of  his  or  her  feelings  of  how  effective  is  the  current  system.  In  this  sense  this 
survey  does  not  intend  to  test  theory  or  find  causal  relations.  Therefore,  a  cross-sectional 
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approach  will  be  useful  in  this  context  (Pinsonneault  and  Kraemer,  1993).  This  approach 
will  allow  the  findings  to  be  generalized  safely  from  the  sample  to  the  population,  and  to 
describe  characteristics  of  the  population  at  the  point  in  time  the  survey  was  conducted. 
After  the  data  collection,  the  data  will  be  analyzed  aiming  to  provide  answers  to  the  three 
research  questions  stated  in  chapter  I.  This  analysis  will  involve  the  development  of 
marginal  and  cross-tabulations  for  the  variables  and  will  make  use  of  simple  descriptive 
statistics  such  as  means,  standard  deviations  and  p-values.  Based  on  the  analysis,  some 
suggestions  will  be  provided  in  order  to  improve  the  quality  of  TMS  in  specific  areas 
detected  by  the  research. 

The  Sample  Population 

Infirstplace,  it  is  necessary  to  identify  the  respondent  (Babbie,  1973).  In  this 
study  the  respondent  will  be  the  end  user  and  it  is  important  to  make  the  distinction 
between  the  end  user  of  information  product  and  the  operator  of  the  program  because  the 
focus  win  be  concentrated  in  the  individual  that  makes  use  of  the  data  and  gets  the 
benefits  of  the  output.  The  target  population  will  be  the  SAO  training  management 
personnel  that  analyze  the  data  from  the  output  reports  and  use  them  as  a  tool  to  manage 
the  training  program.  The  sample  for  this  research  was  composed  of  77  users  located  in 
several  SAO  in  many  countries.  The  sample  frame  was  the  hst  of  about  90  %  of  the  users 
of  TMS  provided  by  the  TMS  supporting  office  located  in  the  Defense  Institute  of 
Security  Assistance  Management  (DISAM)  at  Wright-Patterson  AFB,  Ohio. 
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The  Data  Collection  Procedure 


The  survey  was  sent  by  e-mail  to  all  S  AO  composing  the  sample  frame,  and  the 
responses  were  received  by  e-mail  or  by  fax.  In  the  first  round  29  respondents  sent  their 
evaluations  for  a  response  rate  of  38%.  During  a  follow  on  period  of  one  week,  14  more 
respondents  sent  their  responses  and  the  final  response  rate  was  56%.  The  second  round 
included  the  option  to  send  the  responses  by  fax  in  addition  to  e-mail.  The  response  rate 
of  56  %  was  considered  excellent  for  this  survey  (Pinsonneault  et  al,  1993). 

The  Survey  Instrument 

This  section  describes  how  the  survey  instrument  is  structured  in  terms  of  its 
content,  rating  scales  and  reliability. 

Content.  The  questionnaire  is  divided  in  two  parts.  The  first  part  asks  the 
respondent  about  demographic  data:  length  of  time  using  the  system,  military  or  civilian, 
and  what  version  of  system  is  currently  being  used.  The  second  part  is  composed  of  18 
questions:  13  from  the  short-form  questionnaire  of  Baroudi  and  Orhkowski  (1988)  and  5 
suggested  by  the  sponsoring  agency.  Respecting  to  the  first  13  questions,  the  same 
wording  was  used  with  minor  change  only  in  the  name  of  the  specific  system  being 
evaluated  (TMS)  in  lieu  of  the  general  electronic  data  processing  (EDP).  The 
questionnaire  is  reproduced  in  Appendix  A. 

Rating  Scale.  The  second  part  uses  a  7  point  Likert  scale  for  the  questions^ 

There  are  2  items  per  scale  and  some  of  them  were  reversed  score  in  order  to  minimize 
the  tendency  of  respondents  to  simply  mark  down  one  column. 
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Reliability.  “Reliability  refers  to  the  extent  to  which  the  questionnaire  is  free 
from  measurement  error.  Synonyms  for  reliability  include  dependability,  stability, 
consistency,  predictability,  and  accuracy”  (Stone,  1978).  Utilizing  the  data  collected  in 
the  survey  and  based  on  the  replications  above  described,  the  reliability  for  the  two  items 
of  each  13  scales  and  for  each  three  factors  was  determined  by  calculating  the  Cronbach 
Alpha  coefficient.  The  data  were  processed  using  the  Statistic  Analysis  System  (SAS). 
The  results  for  the  first  case  are  reported  in  Table  1. 

All  the  coefficient  alpha  reliability  estimates  are  above  .60.  The  maximum 
coefficient  is  .98  related  to  the  scale  relevance  of  output.  The  output  data  reported  in 
Appendix  C  were  generated  by  processing  in  the  SAS  program  the  raw  data  reported  in 
Appendix  B.  Those  data  were  utilized  to  build  Table  1. 
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Table  1.  Reliability  Scores 


Scale  Cronbach’s  Alpha 


Relationship  with  TMS  staff 

.83 

Processing  of  requests  for  changes 

.81 

Degree  of  TMS  training  provided 

.73 

1 

i 

User’s  understanding  of  system 

.79 

User’s  sense  of  participation 

.88 

Attitude  of  TMS  staff 

.83 

Reliability  of  output 

.90 

Relevancy  of  output 

.98 

Accuracy  of  output 

.64 

Precision  of  output 

.86 

Communication  with  TMS  staff 

.88 

Time  required  for  new  system  development  | 

1 

.92 

Completeness  of  output 

.82 

The  results  of  reliability  for  each  factor  are  reported  on  the  diagonal  of  Table  2. 
Coefficient  alpha  estimates  were  .86,  .79,  and  .74  for  the  information  product,  TMS  staff 
and  services,  and  knowledge  and  involvement  scales,  respectively.  In  addition,  this  table 
provides  data  for  the  mean  and  standard  deviation  for  each  factor.  This  table  also 
presents  the  correlation  among  the  factors  in  terms  of  Pearson  coefficient.  The  results 
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show  that  there  is  significant  correlation  between  factor  3  and  factors  1  and  2  at  p<.0001. 
Table  2  was  built  by  using  the  output  of  data  processed  reported  in  Appendix  D,  which 
contains  the  correlation  analysis  results  for  the  variables  grouped  in  each  of  the  three 
factors,  and  Appendix  E,  which  contains  the  correlation  analysis  among  the  three  factors 
themselves. 

The  analysis  above  provides  evidence  that  the  questionnaire  is  an  internally 
consistent  measure  and  thus  reasonably  free  of  measurement  error.  It  measures  what  it 
intends  to  measure. 


Table  2.  Means,  Standard  Deviations,  Intercorrelations,  and  Coefficient  Alpha  Rehability 

Estimates  for  the  Study’s  Variables. 


Variables 

Mean 

SD 

1 

2 

3 

1.  Information 
Product 

26.68 

5.88 

(.86) 

2.  TMS  staff 
and  services 

27.47 

4.20 

.30 

(.79) 

3.  Knowledge 
and 

involvement 

18.76 

4.57 

.60** 

.70** 

(.74) 

Note:  N  =  43  Cronbach  alpha  estimates  appear  on  the  diagonal 
**  statistically  significant  at  p<.01 
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Methods  of  Analysis 


The  statistical  method  to  be  used  for  evaluate  differences  in  perceived  satisfaction 
for  the  two  categories  of  users,  military  and  civilian,  will  be  the  pared  t-test.  This 
procedure  will  be  performed  considering  total  satisfaction  and  each  factor:  information 
product,  TMS  staff  and  services,  and  knowledge  and  involvement. 

In  order  to  analyze  the  sensitivity  of  user  satisfaction  with  the  length  of  time  using 
the  system,  it  will  be  used  the  linear  regression  method  for  total  satisfaction  and  each 
factor:  information  product,  TMS  staff  and  services,  and  knowledge  and  involvement. 
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IV.  Results  and  Analysis 


Introduction 

This  section  reports  the  results  of  the  survey  based  on  the  raw  data  processed.  It 
wiU  include  results  from  demographic  data  such  as  distribution  of  years  of  use, 
proportions  between  military  and  civilian  respondents,  and  proportions  of  using  the  two 
versions.  In  addition,  this  section  will  present  the  score  results  for  each  question,  will 
perform  paired  t-test  for  each  factor  considering  the  differences  between  military/civilian 
and  new  version/old  version.  Finally,  it  will  be  reported  the  results  of  regression  analysis 
considering  each  factor  as  the  dependent  variable  and  years  of  use  as  the  independent 
variable. 

Demographic  Data 

The  first  part  of  the  survey  collected  the  following  demographic  data: 

Years  of  Use.  This  question  asked  the  respondent  to  provide  the  number  of  years 
using  the  system.  If  less  than  one  year  is  considered  0  years.  The  results  are  reported  in 
Table  3.  There  are  13  respondents  using  TMS  less  than  one  year. 
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Table  3.  Years  of  Using  the  System 


Years 

percent 

0  (less  than  one) 

13 

30% 

1 

7 

16% 

2 

8 

19% 

3 

9 

21% 

4 

4 

9% 

5  (or  more) 

2 

5% 

Civilian  or  Military.  The  results  for  this  question  showed  that  20  mihtary 
personnel  and  22  civilians  responded  the  question.  There  was  one  non  respondent.  This 
data  resulted  in  47%  of  the  respondents  were  military,  51%  civilian  and  2%  did  not 
respond. 

Version  3.1  or  4.0.  This  question  asked  the  respondent  to  inform  what  version  he 
or  she  uses  currently.  12  use  version  3.1, 28  use  version  4.0,  and  3  did  not  respond.  This 
data  resulted  in  28%  of  the  respondents  use  version  3.1, 65%  use  version  4.0  and  7%  did 
not  respond. 

Score  Results 

The  mean  and  standard  deviation  for  each  question  from  the  standard 
questionnaire  was  calculated  and  reported  in  Table  4.  The  range  for  the  mean  goes  from 
the  minimum  score  4.37  for  processing  of  requests  for  change  to  the  maximum  score 
6.29  for  attitude  of  TMS  staff.  This  latter  scale  also  presented  the  minimum  standard 
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deviation  .93.  The  maximum  standard  deviation  was  1.78  and  was  reported  by  the  scale 
degree  of  TMS  training  provided.  The  total  average  satisfaction  was  calculated  and 
scored  5.27  in  a  scale  from  1  to  7.  Table  4  was  built  by  using  the  raw  data  in  Appendix  B 
and  processed  in  the  spreadsheet  program  Excel. 

Also  the  mean  and  standard  deviation  for  each  added  question  proposed  by  the 
sponsoring  agency  (14,15, 16, 17,  and  18)  was  calculated  and  reported  in  Table  5.  The 
range  for  the  mean  goes  from  the  minimum  score  4.98  for  completeness  of  user 
documentation  to  the  maximum  score  6.24  for  reliability  of  information  provided  by 
TMS  staff.  Also  that  first  scale  presented  the  maximum  standard  deviation  of  1.74  and 
this  latter  scale  presented  the  minimum  standard  deviation  of  .86  showing  more 
agreement  among  the  surveyed.  Table  5  was  built  by  using  the  raw  data  in  Appendix  B 
and  processed  in  the  spreadsheet  program  Excel. 
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Table  4.  Average  Satisfaction  Scores  for  Standard  Questionnaire 


Question  mean  SD 


Relationship 
with  TMS  staff 

5.99 

1.11 

Processing  of 
requests  for 
changes 

4.37 

1.52 

Degree  of  TMS 

training 

provided 

4.44 

1.78 

User’s 

understanding 
of  system 

4.84 

1.66 

HH 

5.12 

1.69 

Attitude  of 

TMS  staff 

6.29 

.93 

5.32 

1.66 

5.97 

1.33 

5.42 

1.50 

5.06 

1.61 

Communication 
with  TMS  staff 

6.05 

1.05 

Time  required 
for  new  system 
development 

4.78 

1.35 

Completeness 
of  output 

4.89 

1.69 

Total  average 

5.27 

1.45 
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Table  5.  Average  Satisfaction  Scores  for  Added  Questions 


Question 

Mean 

SD 

Reliability  of  information 
provided  by  TMS  staff 

6.24 

.86 

Relevancy  of  information 
Provided  by  TMS 

5.99 

1.20 

Accuracy  of  information 
provided  by  TMS  staff 

5.62 

1.51 

Precision  of  information 
provided  by  TMS  staff 

5.64 

1.35 

Completeness  of  user 
documentation  provided 
with  TMS  program 

4.98 

1.74 

Total  average 

5.69 

1.33 

T-test  for  Total  Satisfaction  and  Each  Factor 

The  p- values  for  the  paired  t-test  are  reported  in  Table  6.  This  table  was  built  by 
using  the  data  processed  in  S  AS  program  and  reported  in  Appendix  F.  They  were 
calculated  comparing  the  scores  rated  by  the  categories  military  versus  civilian,  and  old 
version  users  versus  new  version  users.  The  comparison  was  done  for  total  user 
satisfaction  by  summing  the  scores  on  the  13  scales.  For  each  factor  the  scores  were 
calculated  summing  the  scales  loading  in  that  factor.  The  results  show  that  the  difference 
in  scores  rated  by  the  new  and  old  version  users  with  respect  the  quality  of  information 
product  are  statistically  significant  at  p<  .01.  In  addition,  the  variances  for  this  test  are 


37 


significantly  unequal  at  p<  .01.  It  was  not  detected  statistically  significant  difference  for 
the  other  two  factors  and  the  variances  are  significantly  equal  at  p<.01  according  to  the 
data  processed  and  reported  in  Appendix  F.  Also,  no  significant  difference  exist  for  the 
professional  categories  respect  to  each  factor.  Table  6  also  reports  the  average  scores  for 
total  satisfaction  and  each  factor.  The  factor  TMS  staff  and  services  got  the  highest  score 
of  5.50  and  the  factor  knowledge  and  involvement  got  the 
least  score  of  4.69.  Information  product  got  the  score  5.33. 

Table  6.  P-values  for  paired  t-test 


Variable  Military/civilian  version  old/new  avg.  score 


Total  satisfaction 

.307 

.133 

5.27 

Information  Product 

.580 

.001** 

5.33 

TMS  staff  and 
services 

.194 

.888 

5.50 

Knowledge  and 
involvement 

.243 

.278 

4.69 

**  statistically  significant  at  p<  .01 

A  further  one  tail  t-test  was  performed  assuming  unequal  variances  and  version 

3.1  users  reported  a  mean  score  greater  than  version  4.0  users  statistically  significant  at 

p<.001.  The  results  for  this  t-test  are  reported  in  Table  7. 

Table  7.  T-Test  for  Two-Sample 
Assuming  Unequal  Variances 
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Version  3.1  Version  4.0 


Mean 

29,837083 

24,982143 

Variance 

6,8057748 

40,545966 

Observations 

12 

28 

Hypothesized 

Mean 

Difference 

0 

Df 

38 

tStat 

3,4199746 

P(T<=t)  one- 
tail 

0,0007548 

T  Critical  one- 
tail 

1,6859531 

P(T<=t)  two- 
tail 

0,0015097 

t  Critical  two- 
tail 

2,0243942 
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It  was  performed  a  regression  analysis  between  average  satisfaction  as  the 


dependent  variable  and  years  of  using  TMS  as  the  independent  variable.  The  results  are 
reported  in  Table  8  for  total  average  satisfaction  and  for  each  factor  separately.  The  beta 
coefficients  and  p- values  were  calculated  using  the  raw  data  of  Appendix  B  and 
processed  in  Excel.  The  data  processed  are  reported  in  the  Appendix  G  and  were  utilized 
to  built  Table  8.  The  results  show  that  there  is  a  statistically  significant  positive 
correlation  at  p<.05  between  TMS  staff  and  services,  and  years  of  using  the  system. 


Table  8.  Regression  Analysis  year  x  average  satisfaction 


Variable 

Beta  coefficient 

p-value 

Total  Satisfaction 

.14 

.104 

Information  Product 

.13 

.302 

TMS  staff  and  services 

.20 

.014* 

Knowledge  and 
-igynlYement _ 

.22 

.054 

*  statistically  significant  at  p<  .05 

These  results  can  also  be  visualized  in  the  graphic  of  Figure  3.  This  graphic 
shows  the  effect  of  the  positive  correlation  and  the  increase  of  average  satisfaction  with 
TMS  staff  and  services  as  the  time  of  using  the  system  increases. 
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The  beta  coefficient  indicates  that  the  average  satisfaction  with  TMS  staff  and 
services  increases  at  an  average  rate  of  .20  each  year. 
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Figure  4.  Linear  Regression  Plot 
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V.  Conclusions  and  Recomendations 


This  chapter  is  divided  in  two  sections.  The  first  section  draws  the  conclusions 
based  on  the  results  previously  analyzed  in  chapter  IV.  The  second  section  addresses 
some  practical  recommendations  to  the  support  staff  of  TMS,  based  on  the  conelusions. 

Conclusions. 

The  objective  of  this  section  is  to  answer  the  research  questions  stated  in  the 
beginning  of  this  research,  based  on  the  results  analyzed  in  the  previous  chapter. 

Question  1.  What  is  the  system  effectiveness  regarding  to  the  level  of  product 
quality  provided  by  TMS? 

In  order  to  answer  this  question  we  need  to  report  the  data  analyzed  related  to  the 
specific  factor  information  product.  This  factor  relates  to  the  scales  measuring 
reliability,  relevaney,  accuracy,  precision  and  completeness  of  output  information 
provided  by  TMS.  The  average  combined  score  for  these  scales  can  provide  a  good 
indicative  of  customer  satisfaction  with  the  quality  of  product  information  provided  by 
TMS.  According  to  the  previous  results  the  average  score  for  this  factor  is  5.33  in  a  scale 
from  1  to  7.  This  means  that  the  product  information  was  rated  above  the  average  and 
therefore  it  is  a  very  good  quahty  product  under  the  view  of  the  user.  Another  aspeet  to 
be  addressed  is  the  statistically  significant  difference  pointed  by  the  results  related  to 
versions  3.1  and  4.0. 

Aeeording  to  the  results,  users  of  version  3.1  rated  this  version  higher  than  users 
of  version  4.0.  In  general,  it  is  expected  the  user  be  more  satisfied  with  new  versions 
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because  new  versions  correct  any  previous  flaws  of  older  versions.  For  example,  new 
versions  sometimes  bring  improvements  in  user-software  interfaces.  In  order  to  explain 
the  possible  reasons  for  this  outcome  it  is  necessary  to  report  that  28  users  are  using 
version  4.0.  In  addition,  according  to  the  data  reported  in  Table  3, 13  users  of  the  total 
surveyed  are  using  TMS  for  less  than  one  year.  Since  version  4.0  was  fielded  in  less  than 
one  year  (spring  of  1996),  it  is  likely  that  approximately  46%  (13/28)  of  users  of  version 
4.0  have  used  only  this  version.  The  other  54%  of  users  of  version  4.0  have  used  version 
3.1  before  and  were  able  to  compare  the  quality  of  the  two  versions  respecting  to  product 
information,  therefore  rating  version  4.0  less  than  they  would  have  rated  version  3.1. 

Question  2.  What  is  the  level  of  involvement  and  knowledge  of  TMS’  user 
related  to  the  information  services  function? 

This  factor  relates  to  issues  in  processing  of  requests  for  changes  in  the  system, 
degree  of  training  provided  to  the  users,  user’s  understanding  of  TMS,  and  user’s  feeling 
of  participation.  According  to  the  results  the  average  score  for  this  factor  was  4.69.  This 
opinion  is  shared  by  both  military  and  civilian  users,  and  also  by  users  of  version  3.1  and 
4.0.  This  outcome  results  from  the  fact  that  there  was  not  significant  difference  among 
them  with  respect  to  this  factor.  This  means  that  in  a  general  sense,  there  is  enough  room 
for  improvements  in  this  area  and  some  practical  recommendations  can  be  addressed  in 
this  area. 

Question  3.  What  is  the  level  of  user  perceived  satisfaction  with  the  staff  and 
service  provided  by  support  people  of  TMS? 
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More  specifically,  the  scales  utilized  to  measure  this  factor  are:  relationship  with 
the  DISAM  TMS  support  team,  processing  of  requests  for  changes  in  TMS,  perceived 
attitude  of  the  TMS  support  staff,  communication  with  the  TMS  support  staff,  and  time 
required  for  development  of  version  4.0. 

The  average  score  for  this  factor  was  5.50  which  is  a  very  good  score,  and  in 
addition  it  is  the  highest  among  the  three  factors.  Also,  it  is  important  to  note  that  there  is 
a  common  sense  pointing  to  this  opinion.  There  is  no  significant  difference  among 
military  and  civilians  as  well  as  among  users  of  version  3.1  and  users  of  version  4.0  with 
respect  the  perceived  satisfaction  with  TMS  staff  and  services  provided. 

Question  4.  What  is  the  perceived  difference  in  levels  of  satisfaction  between 
military  and  civilian  for  each  of  the  questions  1,2  and  3  above? 

According  to  the  results,  there  was  no  statistically  significant  difference  between 
the  professional  category  respect  to  all  the  three  factors.  It  means  that  professional 
category  (civilian/military)  has  no  influence  on  the  level  of  satisfaction  for  the  three 
factors  considered. 

Question  5.  What  is  the  impact  of  experience  with  the  system  on  questions  1  to  3 

above? 

The  results  show  that  satisfaction  with  information  product  provided  by  TMS 
does  not  depend  on  the  years  of  usage.  This  conclusion  results  from  the  linear  regression 
analysis  performed,  which  showed  no  significant  beta  coefficient  for  this  factor. 

Also,  according  to  the  results  of  regression  analysis,  knowledge  and  involvement 
are  not  dependent  on  the  time  of  experience  the  user  has  in  the  system.  In  other  words  a 
new  user  tends  to  present  the  same  level  of  perceived  satisfaction  with  this  factor  as  an 
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old  user  presents.  This  outcome  results  from  the  fact  that  the  beta  coefficient  is  not 
statistically  significant  at  p<.05  for  this  factor.  Therefore  it  is  possible  to  conclude  that 
the  perceived  satisfaction  in  relation  to  knowledge  and  involvement  has  not  changed  with 
the  time.  This  conclusion  supports  the  fact  that  it  is  necessary  to  perform  improvements 
in  this  area. 

Finally,  the  regression  analysis  presented  a  significant  positive  coefficient  beta  for 
the  factor  TMS  staff  and  services  versus  time.  It  means  that  the  perceived  satisfaction 
with  TMS  staff  and  the  services  provided  tends  to  increase  with  the  time  of  experience 
the  user  has  in  the  system.  This  result  can  be  attributed  to  more  involvement  with  the 
TMS  staff,  and  increased  knowledge  of  the  services  available  as  the  time  of  experience  in 
the  system  increases. 

Recommendations  for  TMS  Staff 

1,  Following  demonstrated  customer  satisfaction  with  the  older,  TMS 
version  3.1,  it  is  recommended  that  DISAM  establish  a  program  to  identify  those  users  of 
version  4.0  who,  having  used  both  versions,  rated  version  4.0  less  than  they  would  have 
rated  version  3.1.  This  would  enable  DISAM  to  concentrate  on  assisting  those  personnel 
and  thus  assuring  their  acceptance  of  the  TMS  4.X  software  as  TMS  4.1  is  fielded  in  its 
final  form. 

2.  It  is  further  recommended  that  DISAM  adopt  a  numbering  scheme  for  their 
software  versions  in  line  with  that  of  commercial  software  venders.  Thus,  initial  “trial” 
versions  would  be  numbered  as  beta  versions,  not  as  if  they  were  “final”  versions.  This 
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would  help  to  instill  confidence  new  final  versions  of  the  software  by  distinguishing 
between  a  final  version  and  a  “test”  or  beta  version. 

3.  It  is  also  reconunended  that  DISAM  continue  to  include  the  end  user  in 
developmental  efforts,  thus  helping  the  end  user  to  feel  it  is  “his/her”  contributions  that 
are  helping  in  the  software  development. 

4.  It  is  recommended  that  DISAM  aggressively  communicate  to  the  field  the 
status  of  software  development.  Only  in  this  way  will  the  users  understand  what  is 
expected  of  them  and  specifically  which  version  of  the  software  they  should  be  using,  etc. 
Interrelationship  with  the  SAN  should  also  be  emphasized. 

Recommendations  for  Future  Research 

It  is  recommended  for  further  research  that  the  dimensions  not  covered  in  the 
questionnaire  used  in  this  research  project,  namely,  use,  individual  impact,  and 
organizational  impact,  be  investigated  in  a  further  literature  review  in  order  to  develop 
one  instrument  or  to  find  the  availability  of  one  already  validated  involving  the  three  cited 
dimensions  when  considered  jointly  or  separately.  In  addition,  this  instrument  could  be 
used  to  replicate  the  findings  of  this  research  project  and  compare  the  results  under  the 
different  dimensions  in  the  model  proposed  by  DeLone  and  McLean  (1992). 

Summary 

This  research  had  the  objective  to  add  to  the  body  of  knowledge  issues  pertaining 
to  the  effectiveness  of  information  systems,  in  particular  the  Training  Management 


47 


System  of  DISAM.  It  used  an  instrument  already  developed  involving  three  dimensions 
of  the  construct  information  system  success.  The  findings  were  used  to  point  out  areas 
for  improvement  in  effectiveness  of  TMS  and  as  a  result  to  provide  practical 
reconunendations  for  that  improvement. 
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Appendix  A:  Survey 


The  following  13  questions  questionnaire  on  the  use  and  support  of  the  Training  Management 
System  (TMS)  is  being  provided  by  Major  Paulo  Ruy  of  the  Brazilian  Air  Force.  Major  Ruy  is  a  student  at 
the  Air  Force  Institute  of  Technology  and  this  questionnaire  and  his  subsequent  analysis  of  responses  will 
be  a  part  of  his  masters  degree  thesis.  Major  Ruy  is  funded  under  an  FMS  case. 

The  person  completing  the  questionnaire  should  be  the  person  who  is  the  primary  user  of  the  TMS  program 
and,  hopefully,  the  person  who  has  the  most  experience  with  TMS.  Please  complete  the  questionnaire  on¬ 
line  on  the  SAN  (IDSS)  and  send  it  as  an  E-mail  message  to  PRUY  on  the  SAN  (IDSS).  Major  Ruy  is  a 
registered  user  on  IDSS. 

Your  cooperation  in  completing  and  returning  this  questionnaire  to  Major  Ruy  is  greatly 
appreciated.  It  will  obviously  contribute  to  his  successful  accomplishment  of  his  degree  work,  but  in 
addition  we  at  DISAM  will  receive  an  excellent  analysis  of  our  efforts  in  fielding  and  supporting  the  TMS 
software. 


Thank  you, 

Mr.  Charles  Collins 

Associate  Professor,  DISAM 


QUESTIONNAIRE  FOR  MEASUREMENTS  OF  USER  INFORMATION  SATISFACTION 

The  purpose  of  this  study  is  to  measure  how  vou  feel  about  certain  aspects  of  the  Training 
Management  System  (TMS)  and  the  support  you  have  been  provided.  On  the  following  pages  you  will  find 
different  factors,  each  related  to  some  aspect  of  your  TMS  support.  You  are  to  rate  each  factor  on  the 
descriptive  scales  that  follow  it,  based  on  your  evaluation  of  the  factor. 

The  scale  positions  are  defined  as  follows: 

adjective X  1234567  adjective Y 

1-extremely  X 
2~quite  X 

3—  slightly  X 

4— neither  X  or  Y;  equally  X  or  Y;  does  not  apply 

5—  slightly  Y 

6—  quite  Y 

7—  extremely  Y 

The  following  example  illustrates  the  scale  positions  and  their  meanings: 


My  vacation  in  Hawaii  was:  (7 )  restful  1  2  3  4  5  6  7  hectic 

(2)  healthy  1  2  3  4  5  6  7  unhealthy 

According  to  the  responses,  the  person's  vacation  was  extremely  hectic  and  quite  healthy. 
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INSTRUCTIONS 


1.  Place  beside  each  scale  the  number  corresponding  the  position  that  describes  your  evaluation  of  the 
factor  being  judged. 

2.  Judge  every  scale;  do  not  omit  any. 

3.  Chose  only  one  number  for  each  scale. 

4.  Work  rapidly.  Rely  on  your  first  impressions. 

Thank  you  very  much  for  your  cooperation. 


COMPLETE  THE  SPACES 

I  am  using  TMS  for _ years  ( If  less  than  1  year,  put  “0”;  otherwise  put  1,2  3. . .) 

I  am _  (military  - 1 ;  civilian  -  2) 

My  responses  are  based  on  the  use  of  TMS  Version _ (If  3.1  enter  “1”  ;  if  4.0  enter  “2”) 


ANSWER  BASED  ON  YOUR  FEELINGS. 


1.  Relationship  with  the  DIS  AM  TMS  () 

support  staff.  ( ) 

2.  Processing  of  requests  for  changes  () 

in  TMS  () 

3.  Degree  of  TMS  training  provided  to  () 

users.  ( ) 

4.  User's  understanding  of  the  TMS.  ( ) 

{) 

5.  User's  feelings  of  participation.  (  ) 

() 

6.  Attitude  of  the  TMS  support  staff.  (  ) 

() 

7.  Reliability  of  output  information  () 

provided  by  TMS  () 

8.  Relevancy  of  output  information  () 


dissonant  1  2  3  4  5  6  7  harmonious 
bad  1  2  3  4  5  6  7  good 


fast  1  2  3  4  5  6  7  slow 
untimely  1  2  3  4  5  6  7  timely 


complete  1  2  3  4  5  6  7  incomplete 
low  1  2  3  4  5  6  7  high 


insufficient  1  2  3  4  5  6  7  sufficient 
complete  1  2  3  4  5  6  7  incomplete 

positive  1  2  3  4  5  6  7  negative 
insufficient  1  2  3  4  5  6  7  sufficient 


cooperative  1  2  3  4  5  6  7  belligerent 
negative  1  2  3  4  5  6  7  positive 


high  1  2  3  4  5  6  7  low 
superior  1  2  3  4  5  6  7  inferior 


useful  1  2  3  4  5  6  7  useless 
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provided  by  TMS  ( to  intended  function  )  () 


relevant  1  2  3  4  5  6  7  irrelevant 


9.  Accuracy  of  output  information  provided  ( ) 


by  TMS.  ( ) 

10.  Precision  of  output  information  provided  ( ) 

by  TMS.  () 

1 1.  Communication  with  the  TMS  ( ) 

support  staff.  (  ) 

12.  Time  required  for  TMS  4.0  (  ) 

development.  ( ) 

13.  Completeness  of  output  information  (  ) 

provided  by  TMS.  ( ) 

14.  Reliability  of  information  ( ) 

provided  by  TMS  staff.  ( ) 


15.  Relevancy  of  information  by  TMS  staff  ( ) 

() 


16.  Accuracy  of  information  (  ) 

provided  by  TMS  staff  ( ) 

17.  Precision  of  information  (  ) 

provided  by  TMS  staff  ( ) 

18.  Completeness  of  user  documentation  (  ) 

provided  with  TMS  program  (  ) 


inaccurate  1  2  3  4  5  6  7  accurate 
low  1  2  3  4  5  6  7  high 


low  1  2  3  4  5  6  7  high 
definite  1  2  3  4  5  6  7  uncertain 


dissonant  1  2  3  4  5  6  7harmonious 
destructive  1  2  3  4  5  6  7  productive 


unreasonable  1  2  3  4  5  6  7  reasonable 
acceptable  1  2  3  4  5  6  7  unacceptable 


sufficient  1  2  3  4  5  6  7  insufficient 
adequate  1  2  3  4  5  6  7  inadequate 


high  1  2  3  4  5  6  7  low 
superior  1  2  3  4  5  6  7  inferior 

useful  1  2  3  4  5  6  7  useless 
relevant  1  2  3  4  5  6  7  irrelevant 


inaccurate  1  2  3  4  5  6  7  accurate 

low  1  2  3  4  5  6  7  high 

low  1  2  3  4  5  6  7  high 
definite  1  2  3  4  5  6  7  uncertain 

sufficient  1  2  3  4  5  6  7  insufficient 

adequate  1  2  3  4  5  6  7  inadequate 
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Appendix  B:  Raw  Data 


Surv#  year 

mil/civ 

version 

la 

1b 

2a 

2b 

3a 

3b 

1 

2 

1 

2 

6 

6 

4 

4 

2 

2 

2 

3 

2 

1 

6 

6 

4.24 

4.39 

6 

6 

3 

0 

1 

2 

5 

5 

4 

4 

6 

6 

4 

2 

1 

2 

2 

6 

4 

4 

6 

6 

5 

5 

2 

2 

7 

7 

7 

7 

7 

6 

6 

2 

1 

2 

7 

7 

4 

4 

2 

2 

7 

3 

2 

2 

7 

7 

5 

5 

7 

7 

8 

0 

2 

2 

7 

7 

2 

6 

4 

4 

9 

5 

2 

2 

6 

6 

5 

5 

6 

6 

10 

2 

1 

1 

5 

5 

4 

4 

2 

2 

11 

4 

1 

1 

7 

6 

4 

4 

1 

6 

12 

0 

1 

2 

7 

7 

4 

4 

4 

4 

13 

4 

2 

7 

7 

7 

7 

5 

6 

14 

0 

2 

1 

6 

6 

3 

5 

1 

1 

15 

1 

1 

6 

6 

4 

4 

2 

3 

16 

0 

1 

2 

6 

5 

3 

3 

3 

3 

17 

1 

2 

1 

6 

6 

6 

6 

5 

5 

18 

0 

1 

2 

6 

6 

2 

2 

3 

5 

19 

3 

2 

2 

7 

6.07 

5 

6 

2 

2 

20 

0 

1 

1 

7 

7 

6 

5 

6 

5 

21 

0 

1 

1 

7 

7 

4 

4 

5 

5 

22 

0 

1 

1 

4 

4 

4 

4 

6 

6 

23 

0 

1 

2 

4 

4 

4 

4 

2 

6 

24 

0 

4 

4 

1 

1 

6 

5 

25 

0 

1 

2 

6 

7 

6 

7 

6 

6 

26 

1 

2 

2 

6 

7 

1 

1 

2 

2 

27 

1 

2 

2 

7 

6 

3 

5 

2 

6 

28 

3 

1 

1 

7 

7 

6 

6 

7 

7 

29 

0 

2 

2 

6 

7 

6 

2 

6 

6 

30 

2 

2 

1 

6 

7 

6 

6 

4 

4 

31 

3 

2 

2 

4 

4 

4 

4 

4 

4 

32 

3 

1 

2 

7 

7 

2 

2 

6 

6 

33 

1 

1 

2 

5 

6 

5 

4 

3 

2 

34 

3 

2 

2 

5 

6 

4 

4 

6 

6 

35 

4 

2 

2 

7 

7 

6 

6 

5 

5 

36 

4 

2 

2 

7 

7 

6 

7 

4 

4 

37 

3 

2 

2 

6 

5 

5 

6 

7 

7 

38 

1 

2 

2 

6 

5 

3 

3 

4 

3 

39 

2 

1 

2 

6 

7 

2 

6 

3 

6 

40 

2 

2 

2 

7 

7 

4 

4 

7 

1 

41 

2 

2 

1 

6 

7 

6 

6 

4 

4 

42 

3 

2 

1 

4 

4 

4 

4 

3 

4 

43 

1 

1 

2 

4 

5 

3 

4 

4 

4 

Surv#  4a 

4b 

5a 

5b 

6a 

6b 

7a 

7b 

8a 

52 


53 


54 


1  6  6 

2  6  6 

3  6  6 

4  6  6 

5  7  7 

6  7  5 

111 
8  6  6 

9  6  6 

10  3  3 

117  7 

12  6  6 

1  1 

14  7  7 

15  6  6 

16  6  6 

17  6  6 

18  6  5 

19  6  5 

20  1  1 

21  7  7 

22  7  7 

23  6  6 

24  4  4 

25  7  6 

26  7  7 

27  7  7 

28  7  7 

29  5  5 

30  7  7 

31  7  7 

32  7  7 

33  6  5 

34  6  6 

35  6  7 

36  1  1 

37  6  6 

38  6  6 

39  6  6 

40  7  7 

41  7  7 

42  6  6 

43  6  6 


6 

6 

6 

6 

7 

7 

7 

6 

4 

5 

6 
7 
7 
7 
6 
6 
6 
6 

5 
7 
7 

6 

5 
4 
7 
1 
7 
7 

6 
7 

4 
7 
6 

5 
7 
7 

6 
6 
5 
7 
7 

5 

6 


5 

6 

5 

6 
7 
6 
6 
2 
6 

5 
7 

6 
7 
6 
6 
7 
6 
6 
7 
7 
1 
6 

5 
4 
7 
2 

6 
7 
6 
6 

4 
7 

5 

6 
7 
7 
6 

5 

6 
7 
6 
6 
6 


5 

7 

5 

6 
7 
6 
6 
2 

4 

5 

6 
6 
7 
6 
6 
6 
6 

5 
7 
7 
7 

6 
7 

4 
7 

3 
6 
6 
6 
6 
6 
7 

5 

5 

6 
7 
6 
6 
6 
7 
6 
6 
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The  SAS  System  1 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

2 'VAR' Variables:  VIA  VIB 


Simple  Statistics 


Variable 

N 

Mean 

Std  Dev 

Sum 

Minimum 

Maximum 

VIA 

43 

5.9070 

1.1915 

254.0000 

2.0000 

7.0000 

VIB 

43 

6.0714 

1.0326 

261.0700 

4.0000 

7.0000 

The  SAS  System  2 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

Cronbach  Coefficient  Alpha 

for  RAW  variables  :  0.834180 
for  STANDARDIZED  variables:  0.839147 

Raw  Variables  Std.  Variables 

Deleted  Correlation  Correlation 

Variable  with  Total  Alpha  with  Total  Alpha 


VIA  0.722871  .  0.722871 

VIB  0.722871  .  0.722871 


The  SAS  System  3 

13:21  Friday,  April  18, 1997 
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Correlation  Analysis 


Pearson  Correlation  Coefficients  /  Prob  >  IRI  under  Ho:  Rho=0  /  N  =  43 


VIA 

VIB 

VIA 

1.00000 

0.72287 

0.0 

0.0001 

VIB 

0.72287 

1.00000 

0.0001 

0.0 

The  SAS  System  4 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

2 'VAR' Variables:  V2A  V2B 


Simple  Statistics 


Variable 

N 

Mean 

Std  Dev 

Sum 

Minimum 

Maximum 

V2A 

43 

4.2381 

1.5089 

182.2400 

1.0000 

7.0000 

V2B 

43 

4.4974 

1.5315 

193.3900 

1.0000 

7.0000 

The  SAS  System  5 


13:21  Friday,  April  18, 1997 


Correlation  Analysis 


Cronbach  Coefficient  Alpha 

for  RAW  variables  :  0.816746 
for  STANDARDIZED  variables:  0.816799 


Raw  Variables  Std.  Variables 

Deleted  Correlation  Correlation 
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Variable  with  Total 


Alpha  with  Total 


Alpha 


V2A 

0.690331 

0.690331 

V2B 

0.690331 

0.690331 

The  SAS  System 

6 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 


Pearson  Correlation  Coefficients  /  Prob  >  IRI  under  Ho:  Rho=0  /  N  =  43 


V2A 

V2B 

V2A 

1.00000 

0.69033 

0.0 

0.0001 

V2B 

0.69033 

1.00000 

0.0001 

0.0 

The  SAS  System  7 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

2 'VAR' Variables:  V3A  V3B 

Simple  Statistics 

Variable  N  Mean  StdDev  Sum  Minimum  Maximum 

V3A  43  4.3256  1.8480  186.0000  1.0000  7.0000 

V3B  43  4.5581  1.7224  196.0000  1.0000  7.0000 

The  SAS  System  8 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 
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Cronbach  Coefficient  Alpha 

for  RAW  variables  :  0.736877 
for  STANDARDIZED  variables:  0.738029 


Raw  Variables  Std.  Variables 

Deleted  Correlation  Correlation 

Variable  with  Total  Alpha  with  Total  Alpha 


V3A  0.584823  .  0.584823 

V3B  0.584823  .  0.584823 


The  SAS  System  9 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

Pearson  Correlation  Coefficients  /  Prob  >  IRI  under  Ho:  Rho=0  /  N  =  43 


V3A 

V3B 

V3A 

1.00000 

0.58482 

0.0 

0.0001 

V3B 

0.58482 

1.00000 

0.0001 

0.0 

The  SAS  System  10 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

2  'VAR'  Variables:  V4A  V4B 
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Simple  Statistics 


Variable 

N 

Mean 

Std  Dev 

Sum 

Minimum 

Maximum 

V4A 

43 

5.0930 

1.5555 

219.0000 

1.0000 

7.0000 

V4B 

43 

4.5814 

1.7352 

197.0000 

1.0000 

7.0000 

The  SAS  System  1 1 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

Cronbach  Coefficient  Alpha 

for  RAW  variables  :  0.797745 
for  STANDARDIZED  variables:  0.800606 


Raw  Variables  Std.  Variables 

Deleted  Correlation  Correlation 

Variable  with  Total  Alpha  with  Total  Alpha 


V4A  0.667509  .  0.667509 

V4B  0.667509  .  0.667509 


The  SAS  System  12 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

Pearson  Correlation  Coefficients  /  Prob  >  IRI  under  Ho:  Rho=0  /  N  =  43 
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V4A 

V4B 

V4A 

1.00000 

0.66751 

0.0 

0.0001 

V4B 

0.66751 

1.00000 

0.0001 

0.0 

The  SAS  System  13 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

2 'VAR' Variables:  V5A  V5B 


Simple  Statistics 


Variable 

N 

Mean 

Std  Dev 

Sum 

Minimum 

Maximum 

V5A 

43 

5.2558 

1.7333 

226.0000 

1.0000 

7.0000 

V5B 

43 

4.9767 

1.6546 

214.0000 

1.0000 

7.0000 

The  SAS  System  14 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

Cronbach  Coefficient  Alpha 

for  RAW  variables  :  0.877463 
for  STANDARDIZED  variables:  0.877995 


Raw  Variables  Std.  Variables 

Deleted  Correlation  Correlation 

Variable  with  Total  Alpha  with  Total  Alpha 


V5A  0.782524  .  0.782524 

V5B  0.782524  .  0.782524 


61 


The  SAS  System 


15 

13:21  Friday,  April  18, 1997 
Correlation  Analysis 

Pearson  Correlation  Coefficients  /  Prob  >  IRI  under  Ho:  Rho=0  /  N  =  43 


V5A  V5B 


V5A 

1.00000 

0.78252 

0.0 

0.0001 

V5B 

0.78252 

1.00000 

0.0001 

0.0 

The  SAS  System  16 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

2 'VAR' Variables:  V6A  V6B 


Simple  Statistics 


Variable 

N 

Mean 

Std  Dev 

Sum 

Minimum 

Maximum 

V6A 

43 

6.2791 

1.0311 

270.0000 

2.0000 

7.0000 

V6B 

43 

6.3023 

0.8319 

271.0000 

4.0000 

7.0000 

The  SAS  System  17 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

Cronbach  Coefficient  Alpha 

for  RAW  variables  :  0.834130 
for  STANDARDIZED  variables:  0.845261 
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Raw  Variables 


Std.  Variables 


Deleted  Correlation  Correlation 

Variable  with  Total  Alpha  with  Total  Alpha 


V6A  0.731994  .  0.731994 

V6B  0.731994  .  0.731994 

The  SAS  System  18 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

Pearson  Correlation  Coefficients  /  Prob  >  IRI  under  Ho:  Rho=0  /  N  =  43 


V6A 

V6B 

V6A 

1.00000 

0.73199 

0.0 

0.0001 

V6B 

0.73199 

1.00000 

0.0001 

0.0 

The  SAS  System  19 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

2 'VAR' Variables:  V7A  V7B 


Simple  Statistics 

Variable  N  Mean  Std  Dev  Sum  Minimum  Maximum 

V7A  43  2.5581  1.6662  110.0000  1.0000  7.0000 

V7B  43  2.7842  1.6696  119.7200  1.0000  7.0000 

The  SAS  System  20 

13:21  Friday,  April  18, 1997 
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Correlation  Analysis 

Cronbach  Coefficient  Alpha 

for  RAW  variables  :  0.903011 
for  STANDARDIZED  variables:  0.903012 


Raw  Variables  Std.  Variables 

Deleted  Correlation  Correlation 

Variable  with  Total  Alpha  with  Total  Alpha 


V7A  0.823174  .  0.823174 

V7B  0.823174  .  0.823174 


The  SAS  System  21 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

Pearson  Correlation  Coefficients  /  Prob  >  IRI  under  Ho:  Rho=0  /  N  =  43 


V7A 

V7B 

V7A 

1.00000 

0.82317 

0.0 

0.0001 

V7B 

0.82317 

1.00000 

0.0001 

0.0 

The  SAS  System 

22 

13:21  Friday,  April  18, 1997 
Correlation  Analysis 
2 'VAR' Variables:  V8A  V8B 

Simple  Statistics 

Variable  N  Mean  Std  Dev  Sum  Minimum  Maximum 
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V8A 

V8B 


43  2.0465  1.3793  88.0000 
43  2.0000  1.2910  86.0000 


1.0000 

1.0000 


7.0000 

7.0000 


The  SAS  System  23 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

Cronbach  Coefficient  Alpha 

for  RAW  variables  :  0.979902 
for  STANDARDIZED  variables:  0.980996 


Raw  Variables  Std.  Variables 

Deleted  Correlation  Correlation 

Variable  with  Total  Alpha  with  Total  Alpha 


V8A  0.962700  .  0.962700 

V8B  0.962700  .  0.962700 


The  SAS  System  24 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

Pearson  Correlation  Coefficients  /  Prob  >  IRI  under  Ho:  Rho=0  /  N  =  43 

V8A  V8B 

V8A  1.00000  0.96270 

0.0  0.0001 
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V8B 


0.96270 

0.000 


1.00000 

0.0 


The  SAS  System  25 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 


2 'VAR' Variables:  V9A  V9B 


Simple  Statistics 


Variable 

N 

Mean 

Std  Dev 

Sum 

Minimum 

Maximum 

V9A 

43 

5.2326 

1.6596 

225.0000 

1.0000 

7.0000 

V9B 

43 

5.6047 

1.3118 

241.0000 

1.0000 

7.0000 

The  SAS  System  26 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

Cronbach  Coefficient  Alpha 

for  RAW  variables  :  0.647138 
for  STANDARDIZED  variables:  0.659185 
Raw  Variables  Std.  Variables 

Deleted  Correlation  Correlation 

Variable  with  Total  Alpha  with  Total  Alpha 


V9A  0.491630  .  0.491630 

V9B  0.491630  .  0.491630 


The  SAS  System  27 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 
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Pearson  Correlation  Coefficients  /  Prob  >  IRI  under  Ho:  Rho=0  /  N  =  43 


V9A 

V9B 

V9A 

1.00000 

0.49163 

0.0 

0.0008 

V9B 

0.49163 

1.00000 

0.0008 

0.0 

The  SAS  System  28 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

2 'VAR' Variables:  VlOA  VlOB 


Simple  Statistics 


Variable 

N 

Mean 

Std  Dev 

Sum 

Minimum 

Maximum 

VlOA 

43 

5.2326 

1.5093 

225.0000 

1.0000 

7.0000 

VlOB 

43 

4.8837 

1.7073 

210.0000 

1.0000 

7.0000 

The  SAS  System  29 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

Cronbach  Coefficient  Alpha 

for  RAW  variables  :  0.859402 
for  STANDARDIZED  variables:  0.863117 


Raw  Variables  Std.  Variables 


Deleted  Correlation 


Correlation 
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Variable  with  Total  Alpha  with  Total  Alpha 


VlOA  0.759195  .  0.759195 

VlOB  0.759195  .  0.759195 

The  SAS  System  30 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

Pearson  Correlation  Coefficients  /  Prob  >  IRl  under  Ho:  Rho=0  /  N  =  43 


VlOA 

VlOB 

VlOA 

1.00000 

0.75920 

0.0 

0.0001 

VlOB 

0.75920 

1.00000 

0.0001 

0.0 

The  SAS  System 

31 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 
2 'VAR' Variables:  VI 1 A  VI  IB 


Simple  Statistics 

Std  Dev  Sum  Minimum  Maximum 

0.9855  261.0000  4.0000  7.0000 
1.1231  259.0000  3.0000  7.0000 

32 

13:21  Friday,  April  18, 1997 
Correlation  Analysis 
Cronbach  Coefficient  Alpha 


Variable 

N 

Mean 

VI 1 A 

43 

6.0698 

VI  IB 

43 

6.0233 

The  SAS 

System 
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for  RAW  variables  :  0.881243 
for  STANDARDIZED  variables:  0.885444 


Raw  Variables  Std.  Variables 

Deleted  Correlation  Correlation 

Variable  with  Total  Alpha  with  Total  Alpha 


VI 1 A  0.794436  .  0.794436 

VI  IB  0.794436  .  0.794436 


The  SAS  System  33 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

Pearson  Correlation  Coefficients  /  Prob  >  IRI  under  Ho:  Rho=0  /  N  =  43 


VllA 

VllB 

VllA 

1.00000 

0.79444 

0.0 

0.0001 

VllB 

0.79444 

1.00000 

0.0001 

0.0 

The  SAS  System 

34 

13:21  Friday,  April  18, 1997 
Correlation  Analysis 


2 'VAR' Variables:  V12A  V12B 


Simple  Statistics 


Variable 

N 

Mean 

Std  Dev 

Sum 

Minimum 

Maximum 

V12A 

43 

4.8237 

1.3078 

207.4200 

2.0000 

7.0000 

V12B 

43 

4.7316 

1.4146 

203.4600 

1.0000 

7.0000 
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The  SAS  System 


35 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

Cronbach  Coefficient  Alpha 

for  RAW  variables  :  0.925425 
for  STANDARDIZED  variables:  0.926957 


Raw  Variables  Std.  Variables 

Deleted  Correlation  Correlation 

Variable  with  Total  Alpha  with  Total  Alpha 


V12A  0.863858  .  0.863858 

V12B  0.863858  .  0.863858 


The  SAS  System  36 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

Pearson  Correlation  Coefficients  /  Prob  >  IRI  under  Ho:  Rho=0  /  N  =  43 


V12A 

V12B 

V12A 

1.00000 

0.86386 

0.0 

0.0001 

V12B 

0.86386 

1.00000 

0.0001 

0.0 

The  SAS  System  37 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

2 'VAR' Variables:  V13A  V13B 
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Simple  Statistics 


Variable 

N 

Mean 

Std  Dev 

Sum 

Minimum 

Maximum 

V13A 

43 

3.1628 

1.7034 

136.0000 

1.0000 

7.0000 

V13B 

43 

3.0465 

1.6897 

131.0000 

1.0000 

7.0000 

The  SAS 

System 

38 

13:21  Friday,  April  18, 1997 

Correlation  Analysis 

Cronbach  Coefficient  Alpha 

for  RAW  variables  :  0.823849 
for  STANDARDIZED  variables:  0.823865 

Raw  Variables  Std.  Variables 

Deleted  Correlation  Correlation 

Variable  with  Total  Alpha  with  Total  Alpha 


V13A  0.700485  .  0.700485 

V13B  0.700485  .  0.700485 

The  SAS  System  39 

13:21  Friday,  April  18,  1997 


Correlation  Analysis 

Pearson  Correlation  Coefficients  /  Prob  >  IRI  under  Ho:  Rho=0  /  N  =  43 


V13A 

V13B 

V13A 

1.00000 

0.70048 

0.0 

0.0001 

V13B 

0.70048 

1.00000 

0.0001 

0.0 
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Appendix  D:  Correlation  Analysis 


The  SAS  System  112 

14:19  Wednesday,  April  23, 1997 


Correlation  Analysis 

5 'VAR' Variables:  V7  V8  V9  VIO  V13 


Simple  Statistics 


Variable 

N 

Mea 

V7 

43 

5.3288 

V8 

43 

5.9767 

V9 

43 

5.4186 

VIO 

43 

5.0581 

V13 

43 

4.8953 

Std  Dev  Sum 

1.5925  229.1400 
1.3227  257.0000 
1.2861  233.0000 
1.5087  217.5000 
1.5643  210.5000 


Minimum  Maximum 

1.0000  7.0000 

1.0000  7.0000 

2.0000  7.0000 

1.0000  7.0000 

1.0000  7.0000 


The  SAS  System 


113 

14:19  Wednesday,  April  23, 1997 


Correlation  Analysis 


Cronbach  Coefficient  Alpha 

for  RAW  variables  :  0.865249 
for  STANDARDIZED  variables:  0.868381 


Raw  Variables  Std.  Variables 


Deleted 

Variable 

Correlation 
with  Total 

Alpha 

Correlation 
with  Total 

Alpha 

V7 

0.757278 

0.818378 

0.759495 

0.823648 

V8 

0.699722 

0.835082 

0.695753 

0.839681 

V9 

0.697068 

0.836419 

0.694914 

0.839889 

VIO 

0.804756 

0.805579 

0.809130 

0.810837 

V13 

0.509385 

0.883159 

0.512510 

0.883233 

The  SAS  System  114 
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14:19  Wednesday,  April  23, 1997 


Correlation  Analysis 

Pearson  Correlation  Coefficients  /  Prob  >  IRI  under  Ho:  Rho=0  /  N  =  43 


V7 

V8 

V9  VIO 

V13 

V7 

1.00000 

0.0 

0.57332 

0.0001 

0.67268 

0.0001 

0.78741 

0.0001 

0.42340 

0.0047 

V8 

0.57332 

0.0001 

1.00000 

0.0 

0.50976 

0.0005 

0.61814 

0.0001 

0.58567 

0.0001 

V9 

0.67268 

0.0001 

0.50976 

0.0005 

1.00000 

0.0 

0.76329 

0.0001 

0.33888 

0.0262 

VIO 

0.78741 

0.0001 

0.61814 

0.0001 

0.76329 

0.0001 

1.00000 

0.0 

0.41625 
0  .0055 

The  SAS  System  115 

14:19  Wednesday,  April  23, 1997 


Correlation  Analysis 


Pearson  Correlation  Coefficients  /  Prob  >  IRI  under  Ho:  Rho=0  /  N  =  43 


V7 

V8 

V9 

VIO 

V13 

0.42340 

0.58567 

0.33888 

0.41625 

1.00000 

0.0047 

0.0001 

0.0262 

0.0055 

0.0 

The  SAS  System  116 

14:19  Wednesday,  April  23, 1997 
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Correlation  Analysis 


5 'VAR' Variables:  VI  V2  V6  VI 1  V12 


Simple  Statistics 

Variable  N  Mean  Std  Dev  Sum  Minimum  Maximum 


VI 

43 

5.9892 

1.0326 

257.5350 

4.0000 

7.0000 

V2 

43 

4.3678 

1.3976 

187.8150 

1.0000 

7.0000 

V6 

43 

6.2907 

0.8676 

270.5000 

4.0000 

7.0000 

VI 1 

43 

6.0465 

0.9989 

260.0000 

4.0000 

7.0000 

V12 

43 

4.7777 

1.3141 

205.4400 

2.0000 

7.0000 

The  SAS  System  117 

14:19  Wednesday,  April  23, 1997 

Correlation  Analysis 

Cronbach  Coefficient  Alpha 

for  RAW  variables  :  0.790171 
for  STANDARDIZED  variables:  0.806123 


Raw  Variables  Std.  Variables 

Deleted  Correlation  Correlation 


Variable 

with  Total 

Alpha 

with  Total 

Alpha 

VI 

0.589241 

0.745750 

0.613188 

0.761930 

V2 

0.578401 

0.755000 

0.572186 

0.774574 

V6 

0.599352 

0.750408 

0.619234 

0.760044 

VI 1 

0.641889 

0.731685 

0.642756 

0.752651 

V12 

0.516851 

0.773305 

0.510010 

0.793254 

The  SAS  System  118 

14:19  Wednesday,  April  23, 1997 
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Correlation  Analysis 


Pearson  Correlation  Coefficients  /  Prob  >  IRI  under  Ho:  Rho=0  /  N  =  43 


VI 

V2 

V6 

Vll 

V12 

VI 

1.00000 

0.0 

0.37079 

0.0144 

0.61578 

0.0001 

0.47372 

0.0013 

0.41307 

0.0059 

V2 

0.37079 

0.0144 

1.00000 

0.0 

0.35772 

0.0185 

0.59292 

0.0001 

0.44633 

0.0027 

V6 

0.61578 

0.0001 

0.35772 

0.0185 

1.00000 

0.0 

0.52661 

0.0003 

0.38860 

0.0100 

VI 1 

0.47372 

0.0013 

0.59292 

0.0001 

0.52661 

0.0003 

1.00000 

0.0 

0.35469 

0.0196 

The  SAS  System  119 

14:19  Wednesday,  April  23, 1997 


Correlation  Analysis 

Pearson  Correlation  Coefficients  /  Prob  >  IRI  under  Ho:  Rho=0  /  N  =  43 


VI 

V2 

V6 

Vll 

V12 

0.41307 

0.44633 

0.38860 

0.35469 

1.00000 

0.0059 

0.0027 

0.0100 

0.0196 

0.0 

The  SAS  System  120 

14:19  Wednesday,  April  23, 1997 

Correlation  Analysis 
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4 'VAR' Variables:  V2  V3  V4  V5 


Simple  Statistics 


Variable 

N 

Mean 

Std  Dev  Sum 

Minimum  Maximum 

V2 

43 

4.3678 

1.3976 

187.8150 

1.0000 

7.0000 

V3 

43 

4.4419 

1.5894 

191.0000 

1.0000 

7.0000 

V4 

43 

4.8372 

1.5029 

208.0000 

1.5000 

7.0000 

V5 

43 

5.1163 

1.5992 

220.0000 

1.0000 

7.0000 

The  SAS  System  121 

14:19  Wednesday,  April  23, 1997 

Correlation  Analysis 

Cronbach  Coefficient  Alpha 

for  RAW  variables  :  0.740167 
for  STANDARDIZED  variables:  0.738456 


Raw  Variables  Std.  Variables 


Deleted 

Variable 

Correlation 
with  Total 

Alpha 

Correlation 
with  Total 

Alpha 

V2 

0.462432 

0.718331 

0.460148 

0.718377 

V3 

0.486245 

0.708678 

0.482763 

0.705924 

V4 

0.505457 

0.696136 

0.499899 

0.696368 

V5 

0.687553 

0.583906 

0.690313 

0.582958 

The  SAS  System  122 

14:19  Wednesday,  April  23, 1997 

Correlation  Analysis 
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Pearson  Correlation  Coefficients  /  Prob  >  IRI  under  Ho:  Rho=0  /  N  =  43 


V2 

V3 

V4 

V5 

V2 

1.00000 

0.0 

0.29628 

0.0537 

0.28005 

0.0689 

0.52775 

0.0003 

V3 

0.29628 

0.0537 

1.00000 

0.0 

0.37720 

0.0126 

0.47575 

0.0013 

V4 

0.28005 

0.0689 

0.37720 

0.0126 

1.00000 

0.0 

0.52567 

0.0003 

V5 

0.52775 

0.0003 

0.47575 

0.0013 

0.52567 

0.0003 

1.00000 

0.0 
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Appendix  E:  Correlation  Analysis 


The  SAS  System  1 

09:03  Friday,  April  25, 1997 

Correlation  Analysis 

3  'VAR'  Variables:  IPSUM  EDPSUM  KISUM 


Simple  Statistics 


Variable 

N 

Mean 

Std  Dev 

Sum 

Minimum 

Maximum 

IPSUM 

43 

26.6777 

5.8857 

1147 

6.5000 

34.0000 

EDPSUM 

43 

27.4719 

4.2025 

1181 

16.5000 

34.5000 

KISUM 

43 

18.7631 

4.5711 

806.8150  8.5000 

27.0000 

The  SAS  System  2 

09:03  Friday,  April  25, 1997 

Correlation  Analysis 

Pearson  Correlation  Coefficients  /  Prob  >  IRI  under  Ho:  Rho=0  /  N  =  43 


IPSUM 

EDPSUM 

KISUM 

IPSUM 

1.00000 

0.0 

0.29896 

0.0515 

0.60248 

0.0001 

EDPSUM 

0.29896 

0.0515 

1.00000 

0.0 

0.70034 

0.0001 

KISUM 

0.60248 

0.0001 

0.70034 

0.0001 

1.00000 

0.0 
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Appendix  F:  T-Test 


The  SAS  System  135 

14:19  Wednesday,  April  23, 1997 

TTEST  PROCEDURE 


Variable:  TOTAL 

MIL  N  Mean  Std  Dev  Std  Error  Minimum  Maximum 

1  20  67.01100000  10.32632298  2.30903601  43.00000000  87.00000000 

2  22  70.53227273  11.61568918  2.47647325  37.00000000  86.50000000 

Variances  T  DF  Prob>ITI 

Unequal  -1.0400  40.0  0.3046 

Equal  -1.0340  40.0  0.3073 

For  HO:  Variances  are  equal,  F  =  1.27  DF  =  (21,19)  Prob>F  =  0.6101 


The  SAS  System  136 

14:19  Wednesday,  April  23, 1997 

TTEST  PROCEDURE 


Variable:  IP 

MIL  N  Mean  Std  Dev  Std  Error  Minimum  Maximum 


1  20  26.20000000  5.77289492  1.29085855  10.00000000  33.50000000 

2  22  27.23363636  6.18453797  1.31854793  6.50000000  34.00000000 

Variances  T  DF  Prob>ITI 

Unequal  -0.5602  40.0  0.5785 

Equal  -0.5583  40.0  0.5798 

For  HO:  Variances  are  equal,  F  =  1.15  DF  =  (21,19)  Prob>F  =  0.7673 
The  SAS  System  137 
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14:19  Wednesday,  April  23, 1997 


TTEST  PROCEDURE 


Variable:  EDP 

MIL  N  Mean  Std  Dev  Std  Error  Minimum  Maximum 

1  20  26.91100000  3.18986867  0.71327632  23.00000000  34.00000000 

2  22  28.48045455  4.36099620  0.92976751  19.00000000  34.50000000 

Variances  T  DF  Prob>ITl 

Unequal  -1.3393  38.3  0.1884 

Equal  -1.3196  40.0  0.1945 

For  HO:  Variances  are  equal,  F  =  1.87  DF  =  (21,19)  Prob>F  =  0.1757 


The  SAS  System  138 

14:19  Wednesday,  April  23, 1997 

TTEST  PROCEDURE 


Variable:  KI 

MIL  N  Mean  Std  Dev  Std  Error  Minimum  Maximum 

1  20  17.95000000  4.34953113  0.97258473  9.00000000  27.00000000 

2  22  19.62795455  4.78501646  1.02016894  8.50000000  26.50000000 

Variances  T  DF  Prob>ITI 

Unequal  -1.1905  40.0  0.2409 

Equal  -1.1850  40.0  0.2430 

For  HO:  Variances  are  equal,  F  =  1.21  DF  =  (21,19)  Prob>F  =  0.6800 


The  SAS  System  139 

14:19  Wednesday,  April  23, 1997 


80 


TTEST  PROCEDURE 


Variable:  TOTAL 

VERSION  N  Mean  Std  Dev  Std  Error 

1  12  72.76458333  7.87005587  2.27188944 

2  28  66.59839286  11.65687455  2.20294222 

Variances  T  DF  Prob>ITl 

Unequal  1.9485  30.4  0.0606 

Equal  1.6703  38.0  0.1031 

For  HO:  Variances  are  equal,  F  =  2.19  DF  =  (27,1 1)  Prob>F  =  0.1715 

The  SAS  System  140 

14:19  Wednesday,  April  23, 1997 

TTEST  PROCEDURE 

Variable:  IP 

VERSION  N  Mean  Std  Dev  Std  Error 

1  12  29.84500000  2.61623081  0.75524078 

2  28  24.98214286  6.36757141  1.20335789 

Variances  T  DF  Prob>ITI 

Unequal  3.4228  38.0  0.0015 

Equal  2.5399  38.0  0.0153 

For  HO:  Variances  are  equal,  F'  =  5.92  DF  =  (27,1 1)  Prob>F'  =  0.0036 

The  SAS  System  141 

14:19  Wednesday,  April  23, 1997 


81 


TTEST  PROCEDURE 


Variable:  EDP 

VERSION  N  Mean  Std  Dev  Std  Error 

1  12  27.75291667  3.72056475  1.07403453 

2  28  27.56267857  3.95883739  0.74814994 

Variances  T  DF  Prob>ITI 

Unequal  0.1453  22.1  0.8858 

Equal  0.1417  38.0  0.8881 

For  HO:  Variances  are  equal,  F'  =  1.13  DF  =  (27,1 1)  Prob>F'  =  0.8664 

The  SAS  System  142 

14:19  Wednesday,  April  23, 1997 

TTEST  PROCEDURE 

Variable:  KI 

VERSION  N  Mean  Std  Dev  Std  Error 

1  12  19.98458333  3.80031545  1.09705657 

2  28  18.26785714  4.78516640  0.90431145 

Variances  T  DF  Prob>ITI 

Unequal  1.2075  26.1  0.2381 

Equal  1.1003  38.0  0.2781 

For  HO:  Variances  are  equal,  F'  =  1.59  DF  =  (27,1 1)  Prob>F'  =  0.4262 
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Appendix  G:  Regression  Analysis 


SUMMARY  OUTPUT  for 
Information  Product 


Regression  Statistics 


Multiple  R 

0,1609484 

R  Square 

0,0259044 

Adjusted  R 

0,002146 

Square 

Standard  Error 

1,231341 

Observations 

43 

ANOVA 

Df 

SS 

MS 

F  Significan 

ce  F 

Regression 

1 

1,65315012 

1,65315 

1,090324  0,302517 

115 

Residual 

41 

62,16422641 

1,516201 

Total 

42 

63,81737653 

Coefficient 

s 

Standard 

Error 

tStat 

P-vaiue 

Intercept 

5,2174096 

0,287713463 

18,13405 

3,43E-21 

X  Variable  1 

0,1287847 

0,123335041 

1,044186 

0,302517 
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SUMMARY 
OUTPUT  for 
TMS  staff 
and  services 


Regression 

Statistics 


Multiple  R 

0,3725075 

R  Square 

0,1387618 

Adjusted  R 
Square 

0,117756 

Standard 

0,7894659 

Error 

Observations 

43 

ANOVA 

Df 

SS 

MS 

F  Significan 

ce  F 

Regression 

1 

4,117157271 

4,117157 

6,60588  0,013894 

88 

Residual 

41 

25,55351478 

0,623256 

Total 

42 

29,67067205 

Coefficient  Standard  Error 

tStat 

P-vaiue 

Intercept 

5,1351591 

0,184465536 

27,83804 

3E-28 

X  Variable  1 

0,2032389 

0,079075426 

2,570191 

0,013895 
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SUMMARY  OUTPUT  for 
Knowledge  and  involvement 


Repression  Statistics 

Multiple  R 

0,2956685 

R  Square 

0,0874198 

Adjusted  R 

0,0651618 

Square 

Standard  Error 

1,1049144 

Observations 

43 

ANOVA 


Df 

SS 

MS  F  Sipnificance  F 

Regression 

1 

4,79490636 

4,794906  3,92756  0,054231965 

Residual 

41 

50,05426871 

1,220836 

Total 

42 

54,84917507 

Coefficients 

Standard 

/  Stat  P-vaiue 

Error 

Intercept 

4,3031317 

0,258172796 

16,66764  7,26E-20 

X  Variable  1 

0,2193301 

0,11067175 

1,981807  0,054232 
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